When you add new music, photos, or videos to a Synology share, by default, the device will scan the new files, render some new thumbnails and index the file in the appropriate library if applicable. This indexing process is necessary for the files to be shown through the various media services such as Photo Station and Music Station. A common complaint from Synology users is that this indexing or thumbnail conversion process can get stuck.
Unfortunately, at the time of this writing, Synology has not implemented any kind of progress bar or anything to indicate whether the process really is frozen. Depending on the type of media being indexed, it may not in fact be frozen and there are perhaps more files to index than you are aware of.
Here I demonstrate the process to monitor the index progress to see if it is truly frozen, or simply to check the index progress out of curiosity.
Connecting Via SSH
This process assumes some familiarity with command-line tools like telnet. In Mac you merely need to open a terminal window, but in Windows you would have to third-party utility like PuTTY (because Microsoft’s Telnet solution does not support SSH protocol).
Update 08/2015: Windows 10 now includes SSH.
If, for whatever reason the Synology does not respond to the SSH request, make sure SSH is enabled by logging into the DiskStation Manager interface and navigating to Control Panel > Terminal & SNMP > Terminal.
ssh [username]@[synology IP]At this point it may say something about an RSA key fingerprint (particularly if this is the first time you’ve ever connected via SSH) and ask if you’re sure you want to continue connecting. Just say yes. By default, the username and password should be the same as your admin account.
Ask (Query) The Database
psql mediaserver postgresThis will open a PSQL command prompt where you can perform what’s called a SQL query, which is basically a fancy way of saying you’re asking the database a question. SQL stands for Structured Query Language and is the standard language for database interaction.
SELECT COUNT(*) AS video_count FROM video;
SELECT COUNT(*) AS music_count FROM music;
SELECT COUNT(*) AS photo_count FROM photo;
SELECT COUNT(*) AS dir_count FROM directory;
SELECT * FROM music WHERE id = (SELECT MAX (id) FROM music);
SELECT * FROM photo WHERE id = (SELECT MAX (id) FROM photo);
SELECT * FROM video WHERE id = (SELECT MAX (id) FROM video);
SELECT * FROM directory WHERE id = (SELECT MAX (id) FROM directory);The above query is actually 8 separate queries. For the first four it will answer with a table indicating how many photos, videos, songs, and folders have been indexed, followed by a comprehensive list of the most recently indexed file under each category, along with related tagging information (such as file name, format, length, etc). Generally, (at least in my experience) photos will be more numerous so will take the longest. So we can check the progress for just the photos like this:
SELECT COUNT(*) AS photo_count FROM photo;Or if you simply wanted to see the name and location of the photo it most recently indexed, you could use this query:
SELECT path FROM photo WHERE id = (SELECT MAX (id) FROM photo);Note that each query starts with a “SELECT” statement and each query ends with a semicolon (;). Capitalizing the different SQL statements is not necessary, just a force of habit.
Update February 22, 2016: As Sietse noted in the comment section, changes between DSM 5.2 and DSM 6 mean that by default, you can no longer login to the root account directly via SSH. Instead you must login as an admin user and escalate to root using the SUDO command. (e.g. sudo psql mediaserver postgres.
Update March 2, 2018: I hadn’t revisited this subject in some time now but I recently had a power outage that seems to have triggered some kind of full reindex of Photo Station.
I had uploaded a bunch of photos and music and nothing had shown up for a couple of hours, which indicated to me that the index was busy doing something else.
For reference, /var/spool/syno_indexing_queue is where all file changes are queued up for the indexer to process. Once the indexer triggers, it moves this file to sync_indexing_queue.tmp, so that’s the file to check when you want to see what it’s currently working on.
When I ran:
cat syno_indexing_queue.tmpIt showed:
#offset:128;
PhotoStation:1:R:photoSomething told DSM to run a full reindex of Photo Station. I’m not sure if it was a coincidence or if it really was the power outage — I’ve never got a straight answer about what exactly triggers a full index, though I’ve seen it more than once. The power outage seems less likely though, since it was shut down smoothly by my UPS controller.
In any case, it got my attention again.
The SQL queries I wrote about before are all well and good, but sometimes I just want a continuous poll of where it’s at, without having to manually run the command over and over. I could do some kind of cron job and output to an HTML file with an auto refresh but it comes up rarely enough that I just wanted a quick & dirty shell script to do it.
Long story short, here’s what I came up with:
while sleep 30; do ls -l /proc/$(pidof synomkthumbd)/fd | grep volume ; doneBasically what the above one-liner does is every 30 seconds it’ll poll the synomkthumbd process and find out what it’s working on and print the results to the console. A new line will output every 30 seconds until you exit the command. the grep volume section at the end just cleans up the output so that it only shows you the relevant information (assumes volume is in the response line as in volume1, volume2 etc).
The command may take a bit to show results because there are times when it’ll be between files so it won’t have any output. Just wait a minute or two and you should start to see results. Or alternatively you could shorten the polling period from 30 to something shorter.
Is your index frozen? Learn how to restart it.
Great tips, thank you!
On my system, it seems that the currently processed file does change, but the number of all entries (photos) in the table is 5000something, while the number of pictures in my original directory is beyond 11000. Maybe it’s just processing it in batches, I’ll check again in the morning.
Just a minor nit-pick:at least on my browser (46.0.2490.71 (64-bit) on Linux), the text color is light grey on white, which is not very readable.
Best regards,
alex.
You’re right it does happen in batches. And in fact there are scenarios where it starts over and instead of indexing new files it confirms that the database is accurate by parsing the existing files. Cases such as adding the Media or iTunes server will prompt this.
You’re right, the text box is a little light. I’ll see what I can do about darkening it. Thanks for the input!
Hello
Thanks for the great tip !
Quick question for you, my MAX ID shows 2464164 whereas I only have about 40000 photos in the queried folder.
I have moved photos from one folder to another a few times, reorganizing things, prior to making a new thumbnails generation. Most of these photos had already been indexed & thumbnailed in the past, so I’m wondering how the ID sequence has reached a high number.
Do you think there is something abnormal with this max ID, might I have a problem with the DB?
It happens that the conversion process is taking weeks and weeks to complete, and I’m beginning to be worried there could be a problem with it.
Thanks
Marc
Great question. The “ID” as far as the database is concerned is merely an arbitrary reference. It’s a number that’s unique for each entry.
For example if you have a photo called DSC_1345.jpg it will get added to the index/database but whenever you share or comment or tag that photo (in Photo Station), the database doesn’t refer to the file name (which is not necessarily unique in the database) but instead uses the unique ID.
The Synology database doesn’t “start over” when reindexing or following upgrades. Rather, it takes the existing database, and grabs the next “ID” available for each subsequent entry. Sometimes upgrades in Photo Station will trigger a flush/reindex which will start not at an ID of 0 or 1, but rather the next highest number already available. This is why with only 40,000 photos you have an ID count of almost 250,000 – this reindex process has probably happened in the background several times over its lifetime. If it were to happen again today due to an update in Photo Station or whatever other reason, the max ID may then be 290,000.
The “max (id)” part of the query above wasn’t so much to retrieve the ID number which is basically irrelevant to us humans – it was simply meant to retrieve the record of the last indexed item which would indicate progress (as opposed to a frozen index process).
The latter query should give an accurate photo count:
SELECT COUNT(*) AS photo_count FROM photo;
However, this query isn’t enough to indicate progress, as a “reindex” process is going through all of the existing photos as well, so the count will only change if new photos are discovered.
Hi,
I get the an error: psql: FATAL: Peer authentication failed for user “postgres”
When i use the command: psql mediaserver postgres
what am i doing wrong?
If I were to guess, I’d say you musr be running DSM Beta 2. They recently beefed up the security to make it harder to accidentally break something.
All that means is now you have to “sudo” the command:
sudo psql mediaserver postgresHi, thats a good guess. :-)
I recently upgrade to Beta 2.
Sudo did it! thanks for the quick reply
Thanks for your excellent description. It definitely did the trick.
If you want to spy on your indexing process on a regular basis, you can set up a small cgi script for that:
In the folder “/usr/syno/synoman/webman” create a file “indexstatus.cgi” with the following content:
#!/bin/ash
echo Content-Type: text/plain
echo “”
psql mediaserver postgres -c “SELECT ‘Current folder: ‘ || path as photostatus FROM directory WHERE id = (SELECT MAX (id) FROM directory) union SELECT ‘Current image: ‘ || title FROM photo WHERE id = (SELECT MAX (id) FROM photo) union SELECT ‘Image count: ‘ || CAST(COUNT(*) as text) FROM photo union SELECT ‘Current song: ‘ || path FROM music WHERE id = (SELECT MAX (id) FROM music) union SELECT ‘Song count: ‘ || CAST(COUNT(*) as text) FROM music”
Then make it executable with “chmod +x indexstatus.cgi” and now you can easily check the status with your browser:
http://DISKSTATION:5000/webman/indexstatus.cgi
Hi, is there any way to run this script from the https connection (port 5001), or from the DSM UI, as I don’t have external http access to my diskstation (http points to another server)
Thanks!
RP
Easiest way would be to forward another port to port 80. So although your DiskStation may only be accessible externally using 5000/5001, and 80 to another computer, you could forward port 8080 or some other random port to the Synology port 80. So the result example would be:
http://youripaddress:8080 or http://yourdomainname:8080 (or other random port).
I copy pasted the above script (shown below) and get the following error – where did I go wrong?
root@DSM2:/usr/syno/synoman/webman# ./indexstatus.cgi
Content-Type: text/plain
“”
./indexstatus.cgi: line 5: syntax error near unexpected token `(‘
./indexstatus.cgi: line 5: `psql mediaserver postgres -c “SELECT ‘Current folder: ‘ || path as photostatus FROM directory WHERE id = (SELECT MAX (id) FROM directory) union SELECT ‘Current image: ‘ || title FROM photo WHERE id = (SELECT MAX (id) FROM photo) union SELECT ‘Image count: ‘ || CAST(COUNT(*) as text) FROM photo union SELECT ‘Current song: ‘ || path FROM music WHERE id = (SELECT MAX (id) FROM music) union SELECT ‘Song count: ‘ || CAST(COUNT(*) as text) FROM music”’
root@DSM2:/usr/syno/synoman/webman# cat indexstatus.cgi
#!/bin/ash
echo Content-Type: text/plain
echo “”
psql mediaserver postgres -c “SELECT ‘Current folder: ‘ || path as photostatus FROM directory WHERE id = (SELECT MAX (id) FROM directory) union SELECT ‘Current image: ‘ || title FROM photo WHERE id = (SELECT MAX (id) FROM photo) union SELECT ‘Image count: ‘ || CAST(COUNT(*) as text) FROM photo union SELECT ‘Current song: ‘ || path FROM music WHERE id = (SELECT MAX (id) FROM music) union SELECT ‘Song count: ‘ || CAST(COUNT(*) as text) FROM music”
Hi,
I followed your instructions and created the script exactly as above – except for the necessity of preceding each command with sudo, everything worked perfectly.
The problem is I can’t access the script from the browser, whether I use
http://(mydiskstation):5000/webman/indexstatus.cgi or with https and 5001, I get a “The page you’re looking for cannot be found” error.
I checked that the file is executable (it appears in green in Putty), and if I run another script that’s in the same directory (like index.cgi or logout.cgi), it works fine. The only difference is that the other scripts are owned by system, while mine is owned by root.
Any ideas? Thanks for your work, you provide some great info!
RP
Hi,
I’m having a problem with indexing since I upgraded to DSM 6, and I’m at my wits end so I’m trying here since you seem to be one of the few who know what they’re talking about!
The symptoms:
– I have no more music files appearing in audio station (on a total of 18000). Audio station reports “There are no music files – copy some with file station etc..”
– I still have access to the photos & videos from photostation and videostation, but no new files seem to be indexed
– Indexing never stops, I always have the message “indexing multimedia files”, but the CPU load is low (5-15% max)
What I tried:
– Restarting the diskstation of course (it’s a 415+)
– Removing all indexed folders from the control panel multimedia indexing applet, then just put music back. after 24hrs nothing happens, but I still have access to my photos and videos- nothing changed there, it just won’t take in new files (although they should disappear if I remove the folders from indexing, right?).
– Tried the command-line fix described in your other post after logging in as root (rm syno_indexing_queue, then kill the processes; restart indexing: no change
– Tried the commands above, and curiously I get a count of 0 for all 3 file types, so I’m wondering if the sql database name hasn’t changed with DSM V6 ?
Anyway, I have the impression that the indexing process doesn’t work sotr some reason, and killing and restarting it won’t fix it. Any other suggestions, of how I could start with a clean slate? Is there some way to delete the database and restart from scratch?
Thanks for any help,
RP
First off thanks for the compliment!
I’m curious how long ago you upgraded to DSM 6. If it was only in the last couple of days, I’d say give it more time – several days. The index process is long and I’m pretty sure music is at the end of the queue – it will go through every last file – even ones that have already been indexed, before finally getting to the new stuff. Depending on your index settings of other file types, it might take a while to get to them. I think it takes my DS1815+ about a week to get through it all.
I can assure you that the database name and general structure is the same as in DSM 5.x.
The indexing process is pretty good, but it’s not perfect. For example let’s say that you deleted a bunch of music while it was occupied indexing videos. For every file you upload or delete, it goes in an indexing queue that tells the indexer what to do with it – to add, or remove it. But then let’s say the indexer appears to be stuck, so you delete the queue (the one from /var/spool) – then no amount of “reindexing” will remove it from the index – at this point the index has fallen out of sync, unless you were to delete and re-add the file. Otherwise would have to remove it manually using the command line, or blow it up and start from scratch. This is a pain and I’ve done it to myself more than once.
Since you’ve already rebooted and cleared the queue, and the index claims to be in progress after you’ve killed it and started it again, I’d say you most likely just have to wait longer.
I can share with you some commands that I find useful:
sudo ls -l /proc/$(pidof synomediaparser)/fd
sudo ls -l /proc/$(pidof synoindexd)/fd
The two commands above will basically output a list of files that the indexer or the media parser are currently accessing. I sometimes have to run the same command 10 times before it shows me a media path. I use it to gauge what exactly the indexer is working on, in the event I grow impatient.
If, after all your patience, it still seems that something is wrong with the index database, there is a method of last resort. And of course I would advice extreme caution in this case, and I take no responsibility for those who do this:
rm -rf ‘/volume1/@database/pgsql’
This will delete the postgres database that runs most of the core Synology services. Photo Station, Video Station, the Downloader, and a handful of other applications store their configuration in the same database, so if you delete the database you’ll lose the configuration of those applications as well, including any files you’re currently downloading in Download Station. However, upon reboot, a new database will be generated by the startup script. I must reiterate that any customizations that you’ve done in Photo or Video Station – photo album customizations, comments, etc. – will disappear.
Hi Andrew,
First thanks for the long and involved reply, and sorry for my tardiness but I had put an email alert on this thread so I was obliviously waiting to be notified if something happened!
I followed your advice, and just waited. After an inordinate amount of time (several days), all the music and photos ended up by being indexed correctly – I gave up on the videos, as I’m using another media server anyway (Serviio), so I don’t much care about those anyway.
So all was going well until a couple days ago, where I added a half dozen new albums to music, and they never showed up (hence my new visit to this post!).
I looked up in the CP: “Indexing media files…”. I went to ssh, looked up the Db which showed the last music file added on may 18th (new ones I’m talking about were added on the 23rd).
Seems the indexing is stuck again, so I rebooted DSM, killed and restarted the process, and we’ll see what happens.
Still, all this seems like one or more bugs to me. They should fix it, and put a bit more UI on the whole process as a courtesy to the users, starting with a progress bar and some feedback on files already indexed…
By the way, I can perfectly login directly as root in Windows using Putty, I don’t have to go through Admin for this.
Thanks again for the help,
RP
Update to the above: 1 hour after deleting syno_index_queue and restarting the indexing, the file is still at its creation timestamp, with 6 bytes in it: R:all. So it’s stuck again, and I’ll be damned if I know why.
It’s really a shame, that they did such a good job of their DSM globally, and so grossly botched this part. Just ranting in exasperation!
RP
Actually Syno_index_queue may remain empty – I’m not sure this file does what you think it does.
Let’s say the Synology starts an indexing process. Then let’s say you save a load of new media files to the server in the same places it’s indexing. The queue is the placeholder that will log and list the changes since the indexing process started. It’s not going to show the exhaustive list of files that the indexer is processing, only the ones that weren’t there when it started. Likewise if the indexer is complete, that same file will collect changes and the indexer will regularly check and process that file.
Ok, understood about the queue… But if I look up the database:
psql mediaserver postgres
SELECT * FROM music WHERE id = (SELECT MAX (id) FROM music);
I still get that the last file was indexed on 18th May… and that nothing has moved since then, despite all I did.
What do you think could prevent the process from restarting, despite me killing it, rebooting the Diskstation, etc? Any ideas ?
Thanks!
RP
There’s a lot of variables that I don’t know. That’s the music index, sure, but what’s it indexing in the meantime? Depending on how it’s configured, and whether it’s just started over, it might be going over what’s already been indexed to the database, and hasn’t yet reached the music (again).
I thought I’d written something about this already, but I’m thinking now it was a draft I haven’t finished, outlining how to see exactly what file the indexer is accessing at any given time. It’s a bit clunky, but try this:
sudo ls -l /proc/$(pidof synoindexd)/fd
Basically it will list the files used by the synoindexd process.
You might have to run it a dozen times or so for it to show a file name – it moves fast behind the scenes and most of the time you’ll probably be catching it *between* file access processes. But assuming the index is running, you might be able to gauge whether it’s *moving* by using this command.
You could ask Synology support about it too, if it’s still under warranty. Possibly even if it’s not.
Then there is the nuclear option:
sudo rm -rf /var/database/pgsqlsudo rm -rf /volume1/@database/pgsql
This will delete the PostgreSQL database that indexing and a lot of the apps run on. Just be aware that even apps like Download Manager run on this, and maybe notes. I’m not sure what else. The next time you reboot, it will re-create those databases and start the indexing from *true* zero.
It finally got around to catalog the new music files I had added, after 10 days… No need for the nuclear option after all!
I understand that indexing is a complex process, but still it’s not rocket science and Syno should at least add some visual feedback that the process is going on and doing something. A progress bar would be tricky, but it wouldn’t be so hard to add a counter and the folder/ files being catalogued.
Thanks for your invaluable and informed help anyway, I see there are lots of searches around this indexing feature so I hope it helps other people as well.
RP
Thanks – very helpful. My index had become corrupted. I managed to set a re-index going (via the ‘indexing’ in the control panel interface and then going in via Putty enabled me to see the re-indexing taking place – and work out how long it might take!
Hi, is there a way to list what has been tagged (or left) for indexing? Cheers.
Hi, I have a synology 213J. when i login as root and try commands like psql or even less i get the following. sudo is not even recognized. Do you know if there is some proprietary distribution they are using that hides access to commands like this
SERVER> psql
-sh: psql: not found
SERVER> sudo
-sh: sudo: not found
THanks
I guess first of all it would help to know which version of DSM you’re on. They’ve changed SSH security a couple of times between 5.x and 6.x
If you’re logging in as root then sudo wouldn’t even be necessary. And, if I’m not mistaken, DSM 5.x uses “pgsql” instead of “psql”
Funny LOG in and enable SSH, I cant login !!!!
disk full message when it is false….I a SOL
Andrew many thanks for the post and the detailed explanations,. In conjunction with using my workstation to create the thumbnails it has saved me days and days of waiting!
During the time I did have to wait I played with the output format of SQL you used to monitor the last indexing progress.
SELECT ‘Music’ as Media, Nr.Count, Path as “Last Processed”, Title, Date, MDate FROM music, (SELECT COUNT(*) as Count FROM music) as Nr WHERE id = (SELECT MAX (id) FROM music)
union all
SELECT ‘Photos’ as Media, Nr.Count, Path as “Last Processed”, Title, Date, MDate FROM photo, (SELECT COUNT(*) as Count FROM photo) as Nr WHERE id = (SELECT MAX (id) FROM photo)
union all
SELECT ‘Videos’ as Media, Nr.Count, Path as “Last Processed”, Title, Date, MDate FROM video, (SELECT COUNT(*) as Count FROM video) as Nr WHERE id = (SELECT MAX (id) FROM video)
union all
SELECT ‘Directories’ as Media, Nr.Count, Path as “Last Processed”, Title, Date, MDate FROM directory, (SELECT COUNT(*) as Count FROM directory) as Nr WHERE id = (SELECT MAX (id) FROM directory);
It certainly is more complicated but it does provide all the information in one table!
PS Sorry if there are any typos, I’m finding it quite difficult to read the light grey text on a white background.
I do have one question though. After creating the Thumbnails for 50k photos, I have installed the latest upgrade to DSM upgrade (DSM 6.1-15047 Update 2) and restarted the media server. The Synomediaparser is taking up over 90% of my cpu and appears to be trolling through all my media directories but not updating any of the tables in the DB. Does anyone know what the Synomediaparser process is doing. Hopefully not generating any thumbnails or converted media files, as this is the work I did on my workstation.
No, I think it just gets triggered by the indexer to make sure that there’s matching thumbnails for each photo. Sort of a “double-check” scan.
Hi,
Thanks for the insights into Synology media indexing. I’ve recently had similar bad indexing experiences as other commentators have shared: indexing is slow, it (appears to) hangs, etc.
Using your excellent tips, I’ve monitored and restarted the indexing processes several times but the indexing always grinds to a halt again.
Rather than try exactly the same things over and over again, I decided on an experiment. I have music only on the DiskStation; the media indexing is pointed towards a (network-shared) USB drive attached to the DiskStation. This USB drive has an iTunes library (and other music), that I manage from a Windows PC. In the past, as I added music to the USB drive via iTunes, my Diskstation would index changes to the USB drive, and all was well.
Since my latest DSM 6.1 update (to DSM 6.1-15047 Update 2), media indexing has balked. Using your tips for checking out indexing-process file descriptors and postgres sql statements, the directory- and file-naming conventions iTunes employs were revealed. It occurred to me that media indexing might have problems with some directory names if it wasn’t ‘escaping’ them correctly or completely. (it also occurred to me that indexing was having issues crawling over an attached filesystem, rather than its own internal drives.)
Anyway, I copied all my music files from the USB drive to the Diskstation drive (fun with the ‘find’ command and shell scripts!)
After that, media indexing the diskstation internal drive was quick, with no hangs!
I now have my music library in two places: on the USB drive for iTunes/Windows use, and on my Diskstation internal drive for network audio clients. In a way, each is the backup of the other. And I can use the “-newer” option in my “find” commands to copy new music files from the USB drive to the Internal Diskstation drive.
I just thought I’d report my experience in case it might be useful to anyone else.
Thanks Andrew for these amazing insights into the Synology PhotoStation indexing issues. Is there any plan to provide similar insights for the Synology Moments app given that Synology seem to be pushing users towards that and it appears to use the same or similar indexing routines (along with similar indexing issues!) with different postgres databases/tables?
No immediate plans. Although I like the general interface of Moments, it feels to me like some of the implementation is half-baked. I mean any software has to start somewhere, but to me it lacks all of the features that I currently use it for — namely, a photo album for the whole family to funnel into and share from, rather than a disparate assortment of isolated albums for individual users.
For now I only play with it in my test lab, but until it has feature parity I don’t have much use for it.
Hello Andrew. I’m probably a little impatient, but I figured I would use the information you provided to check on my indexing status to be sure it isn’t stuck. 300 movies on a DS415+ indexing for 24 hours, so far. When I run sudo psql mediaserver postgres, I get a password prompt, which doesn’t match my admin credentials (running version DSM 6.1.6-15266). Any ideas as to where I might find the password? Thank you.
The sudo/root password should match the admin account, so I’m not sure what to tell you there.
24 hours for 300 movies does seem like a long time, but there are other possible factors. For example, something might have triggered a re-index of other data. So it might still be indexing photos or music and the new movies are further down the queue.
If you can’t do the pgsql query, I’d suggest trying the other steps I added on March 2nd this year. Or even to check the syno_indexing_queue and syno_indexing_queue.tmp files and see what else is on the list besides those movies.
Thank you, again, Andrew (especially for the quick reply)! The admin password let me in this time (maybe a typo previously). Have a great night!
Hi,
the “while sleep 30; do ls -l /proc/$(pidof synomkthumbd)/fd | grep volume ; done”
ends up in a permission denied … ligin with the “admin”-user. This one should be root, as I do not need “sudo”s for commands.
On DSM 6.2.xxx the /var/spool/syno_indexing_queue ends up in “Does not exist”
instead a whole bunch of conv_progress_video etc appear. permission denied.
AM I doing domething wrong, or have the guys again schanged a lot?
I am looking for exact what you decribed in the headline AND an indication, how long FILE indexing (no media) takes.
More ideas?
If there is no syno_indexing_queue file then the indexing has completed, at least as far as the service is concerned. If you believe it has not indexed everything, you could trigger a full re-index manually with the following command:
synoindex -R -all
Regarding how long file indexing takes, that depends on how many files there are to index. And depending on what other files come first in the queue. The different types of media indexing happen sequentially, so if the service was still indexing media then it would have to complete that before progressing to regular files.
I did the command: while sleep 30; do ls -l /proc/$(pidof synomkthumbd)/fd | grep volume ; done
I’m getting this every 30 sec: ls: cannot open directory /proc/22363/fd: Permission denied
You have to run it as root. So if you run it manually it would be:
sudo while sleep 30; do ls -l /proc/$(pidof synomkthumbd)/fd | grep volume ; doneExcellent article. Thank you!
Hi Andrew, very nice page/s, thanks a lot for your hints!
I have two questions related to re-indexing at my DS214play (DSM 6.2.2-24922 Update 4):
a) I recognize that “sudo ls -l /proc/$(pidof synomkthumbd)/fd | grep volume” provides changing directories (each taking some time depending on size). Do you consider those changes sufficient to conclude the re-indexing process is running and expected to complete successfully some time? I am asking this, since the postgres approach does not show incrementing numbers at all…
b) “sudo while sleep xx; do ls -l /proc/$(pidof synomkthumbd)/fd | grep volume ; done” does not work for me, unfortunately. It always tells “-sh: syntax error near unexpected token `do'”. Any idea?
Thanks a lot for your advice, and take care!
any way to force export to file.vsmeta?
through the web interface yes, but through ssh?
try:
sudo sh -c ‘while sleep 5; do ls -l /proc/$(pidof synomkthumbd)/fd | grep volume ; done’
(I’m no Linux or Synology expert in anyway, I just puzzled together the stuff that you’re just about to read!)
I had a case when fileindexd (Universal Search) permanently got stuck in a specific directory (restart of the NAS would not help). Removing the offending folder from the list of indexed folders and then killing fileindexd got it unstuck.
I’ve just readded the folder back to the list of indexed folders, we’ll see what happens…
Perform all following commands as root: either prepend them with `sudo` or start a root shell first using `sudo -i`
How to find out which directory fileindexd currently is indexing:
1) Install synogear first:
https://github.com/SynoCommunity/spksrc/wiki/FAQ-synogear
2) Run the following command line to see which directory fileindexd is currently accessing:
lsof -c fileindexd
Repeat the above command several times. If the directory doesn’t change for a long time (>10min), there’s a chance that fileindexd might be stuck.
To find out the process ID to kill:
ps xao pid,comm | grep fileindexd
(alternatively use `htop` to have a look at the running processes)
First try to terminate the process gracefully:
kill -p PID_of_fileindexd
If fileindexd terminated successfully, `ps xao pid,comm | grep fileindexd` will show a new PID since fileindexd will be restarted immediately.
Otherwise, terminate the process forcefully:
kill -9 -p PID_of_fileindexd
Hopefully, using the following command, you should now see fileindexd traversing your volumes:
while sleep 1; do lsof -c fileindexd | grep –color=never -o -e ‘\/volume.*’; done
The behavior that I’m seeing on a healthy process is that fileindexd switches between
a) traversing directories, then
b) doing some heavy processing (high CPU) for a few minutes while not changing directories.
I couldn’t get the looped execution of the process ID to work for me but I used the principles here to write a shell script. It’s designed to check music indexing by default but you can run it passing in “photo”, “video” or “directory” as arguments instead. It will loop track until indexing stops.
#!/bin/bash
#
[[ $1 != “” ]] && MEDIA_TYPE=”$1″ || MEDIA_TYPE=”music”
OUTPUT=”FIRST”
LAST_OUTPUT=”UNSET”
echo “Tracking indexing on $MEDIA_TYPE..”
while [ “$LAST_OUTPUT” != “$OUTPUT” ]
do
LAST_OUTPUT=$OUTPUT
OUTPUT=$(sudo psql mediaserver postgres –command=”SELECT COUNT(*) AS count FROM $MEDIA_TYPE;”)
if [ “$LAST_OUTPUT” != “$OUTPUT” ]
then
echo
echo “Current Total:”
echo $OUTPUT
echo
echo “Indexing status analysing…”
declare -i i=1
while [ $i -le 5 ]
do
LAST_FILE_OUTPUT=$FILE_OUTPUT
FILE_OUTPUT=$(sudo psql mediaserver postgres –command=”SELECT * FROM $MEDIA_TYPE WHERE id = (SELECT MAX (id) FROM $MEDIA_TYPE)” | grep volume)
if [ “$FILE_OUTPUT” != “$LAST_FILE_OUTPUT” ]
then
echo
echo “Last indexed file ($i):”
echo $FILE_OUTPUT
sleep 4
else
sleep 1
echo -n “-”
fi
((i++))
done
echo
echo “———————————”
fi
sleep 4
done
echo
echo “Indexing has completed or stalled”
echo
echo “———————————“
I’ve uploaded the shell script to GitHub here if anyone wants to embellish it any…
https://github.com/tiefling/synology-media-indexing-tracker/blob/main/indexing-monitor.sh