We have a channel which is used to "reassemble" files from a special format. For example, let's say we have a message named "file1.xml" and this file contains several attachments or notes inside of it. This file1.xml is then converted into several different files: LST_file1.xml is like a manifest; it lists all of the other files needed to reassemble the original file. There is one file for the original message with the attachments stripped, and there is then one file for each attachment that we stripped out.
Now, there is a channel that watches a folder for these LST_* files. When a file arrives, it picks it up and sets some channel map variables in the source transformer. Then, the first destination is a JavaScript writer. It passes the LST_* file into a Java function which parses the list files and looks to see if all of the other files needed to reassemble have arrived or not. If they have, the original file is then reassembled in a different directory and the LST and attachment files are deleted. If they have NOT, we leave the file alone and do nothing. The next time the channel cycles it will pick up the same file and again examine it to see if it is ready for reassembly.
Now, there are a number of problems with this: first, if one or more of the constituent files never arrive, the LST_* file stays there forever and is picked up over and over. Secondly, if the LST file is corrupted in some way, it could generate an error and, since the source transformer does not delete or move files, again this happens over and over.
Now, I am making some changes to correct this. First, in my destination transformer I have added a check to compare the lastModifiedDate of the LST_* file against the current time and, if I exceed a threshold I assume the files are never going to arrive and I throw an error, log it, and delete the LST file.
But, for files where we are cycling I think it would be better to use queuing somehow instead of just letting the source connector just keep picking up the same LST file over and over waiting for everything to arrive. At best, this throws the dashboard statistics off since we're bumping up the stats every time the same file gets looked at.
So, what would be a good way to design this channel? Is there a way to re-queue a message from a JavaScript Writer destination transformer? Either re-queue it for the source connector, or just re-queue it for the destination? What I want is for the LST message to be examined and, if all files are not arrived and we haven't exceeded either a number of attempts or an elapsed interval, I want to re-queue it so that I can check again. If I exceed the number of attempts (or interval as I currently have it), then I want to throw an error, call a code template function to log it, and then delete the LST and attachment files.
Now, there is a channel that watches a folder for these LST_* files. When a file arrives, it picks it up and sets some channel map variables in the source transformer. Then, the first destination is a JavaScript writer. It passes the LST_* file into a Java function which parses the list files and looks to see if all of the other files needed to reassemble have arrived or not. If they have, the original file is then reassembled in a different directory and the LST and attachment files are deleted. If they have NOT, we leave the file alone and do nothing. The next time the channel cycles it will pick up the same file and again examine it to see if it is ready for reassembly.
Now, there are a number of problems with this: first, if one or more of the constituent files never arrive, the LST_* file stays there forever and is picked up over and over. Secondly, if the LST file is corrupted in some way, it could generate an error and, since the source transformer does not delete or move files, again this happens over and over.
Now, I am making some changes to correct this. First, in my destination transformer I have added a check to compare the lastModifiedDate of the LST_* file against the current time and, if I exceed a threshold I assume the files are never going to arrive and I throw an error, log it, and delete the LST file.
But, for files where we are cycling I think it would be better to use queuing somehow instead of just letting the source connector just keep picking up the same LST file over and over waiting for everything to arrive. At best, this throws the dashboard statistics off since we're bumping up the stats every time the same file gets looked at.
So, what would be a good way to design this channel? Is there a way to re-queue a message from a JavaScript Writer destination transformer? Either re-queue it for the source connector, or just re-queue it for the destination? What I want is for the LST message to be examined and, if all files are not arrived and we haven't exceeded either a number of attempts or an elapsed interval, I want to re-queue it so that I can check again. If I exceed the number of attempts (or interval as I currently have it), then I want to throw an error, call a code template function to log it, and then delete the LST and attachment files.
Suggestions for how to implement a channel using queueing
0 commentaires:
Enregistrer un commentaire