Merge pull request #218 from sy-c/master

sy-c · web-flow · commit 6d55dc4e88c8 · 2022-04-20T17:07:13.000+02:00
v2.10.1
diff --git a/doc/configurationParameters.md b/doc/configurationParameters.md
@@ -132,10 +132,11 @@ The parameters related to 3rd-party libraries are described here for convenience
 | equipment-dummy-* | eventMinSize | bytes | 128k | Minimum size of randomly generated event. |
 | equipment-dummy-* | fillData | int | 0 | Pattern used to fill data page: (0) no pattern used, data page is left untouched, with whatever values were in memory (1) incremental byte pattern (2) incremental word pattern, with one random word out of 5. |
 | equipment-player-* | autoChunk | int | 0 | When set, the file is replayed once, and cut automatically in data pages compatible with memory bank settings and RDH information. In this mode the preLoad and fillPage options have no effect. |
-| equipment-player-* | autoChunkLoop | int | 0 | When set, the file is replayed in loops. Trigger orbit counter in RDH are modified for iterations after the first one, so that they keep increasing. If value is negative, only that number of loop is executed (-5 -> 5x replay). |
+| equipment-player-* | autoChunkLoop | int | 0 | When set, the file is replayed in loops. If value is negative, only that number of loop is executed (-5 -> 5x replay). |
 | equipment-player-* | filePath | string | | Path of file containing data to be injected in readout. |
 | equipment-player-* | fillPage | int | 1 | If 1, content of data file is copied multiple time in each data page until page is full (or almost full: on the last iteration, there is no partial copy if remaining space is smaller than full file size). If 0, data file is copied exactly once in each data page. |
 | equipment-player-* | preLoad | int | 1 | If 1, data pages preloaded with file content on startup. If 0, data is copied at runtime. |
+| equipment-player-* | updateOrbits | int | 1 | When set, trigger orbit counters in all RDH are modified for iterations after the first one (in file loop replay mode), so that they keep increasing. |
 | equipment-rorc-* | cardId | string | | ID of the board to be used. Typically, a PCI bus device id. c.f. AliceO2::roc::Parameters. |
 | equipment-rorc-* | channelNumber | int | 0 | Channel number of the board to be used. Typically 0 for CRU, or 0-5 for CRORC. c.f. AliceO2::roc::Parameters. |
 | equipment-rorc-* | cleanPageBeforeUse | int | 0 | If set, data pages are filled with zero before being given for writing by device. Slow, but usefull to readout incomplete pages (driver currently does not return correctly number of bytes written in page. |
@@ -148,9 +149,11 @@ The parameters related to 3rd-party libraries are described here for convenience
 | equipment-zmq-* | type | string | SUB | Type of ZMQ socket to use to get data (PULL, SUB). |
 | readout | aggregatorSliceTimeout | double | 0 | When set, slices (groups) of pages are flushed if not updated after given timeout (otherwise closed only on beginning of next TF, or on stop). |
 | readout | aggregatorStfTimeout | double | 0 | When set, subtimeframes are buffered until timeout (otherwise, sent immediately and independently for each data source). |
+| readout | customCommands | string | | List of key=value pairs defining some custom shell commands to be executed at before/after state change commands. |
 | readout | disableAggregatorSlicing | int | 0 | When set, the aggregator slicing is disabled, data pages are passed through without grouping/slicing. |
 | readout | disableTimeframes | int | 0 | When set, all timeframe related features are disabled (this may supersede other config parameters). |
 | readout | exitTimeout | double | -1 | Time in seconds after which the program exits automatically. -1 for unlimited. |
+| readout | flushConsumerTimeout | double | 1 | Time in seconds to wait before stopping the consumers (ie wait allocated pages released). 0 means stop immediately. |
 | readout | flushEquipmentTimeout | double | 1 | Time in seconds to wait for data once the equipments are stopped. 0 means stop immediately. |
 | readout | logbookApiToken | string | | The token to be used for the logbook API. |
 | readout | logbookEnabled | int | 0 | When set, the logbook is enabled and populated with readout stats at runtime. |
diff --git a/doc/memory.md b/doc/memory.md
@@ -0,0 +1,18 @@
+Here is a typical buffer organisation for a production FLP:
+- there is one buffer per CRU end point or CRORC channel, providing empty data pages (superpages) to the hardware for DMA transfer from the readout card to the computer memory.
+- there is one buffer for the FMQ channel to Data Distribution, to copy data which overlap data pages to a single block of memory. Data Distribution requires that data from a HeartBeatFrame (HBF) is shipped in a single FMQ message.
+
+All buffers and data transfer operations are independant one from the other. If a single buffer gets full, the whole FLP system can be affected: CRU packets dropped, incomplete timeframes, data synchronization or consistency issues.
+
+Buffers are circular, pages are used in the order they are put in the buffer. On startup, the pages are in the order of their memory address.
+Buffer and pages size should be adapted to the throughput and data pattern at runtime.
+
+
+In case of some "buffer low" issues, there are 3 log messages for each episode:
+
+1) "buffer usage is high", when reaching 90% usage.
+2) "buffer full" at 100% usage.
+3) "buffer back to reasonable" when down to below 80% usage.
+
+When one of the buffer is full, other messages will start to appear, depending on the context (no page left, packets dropped, etc).
+
diff --git a/doc/releaseNotes.md b/doc/releaseNotes.md
@@ -447,3 +447,7 @@ This file describes the main feature changes for each readout.exe released versi
   - readout.flushConsumerTimeout: when set, readout waits up to this amount of time that all data pages locked by consumers are released before stopping.
 - Added warning message on buffers low.
 - Added message at end of run showing the links which have provided data for each equipment and how much per link.
+
+## v2.10.1 - 20/04/2022
+- Updated configuration parameters:
+  - equipment-file-*.updateOrbits: when set to zero, RDH orbits are not updated in file loop replay. This is needed for some reconstruction tests. This however creates a stream of data with inconsistent orbit ids and mismatching timeframe information.
diff --git a/src/DataBlock.h b/src/DataBlock.h
@@ -62,16 +62,17 @@ struct DataBlockHeader {
   uint32_t timeframeOrbitLast;  ///< from timeframe
   uint8_t flagEndOfTimeframe;   ///< flag to signal this is the last TF block
   uint8_t isRdhFormat;          ///< flag set when payload is RDH-formatted
+  uint32_t orbitOffset;         ///< set when RDH orbits should be added given offset to match TFid
 
   uint8_t userSpace[DataBlockHeaderUserSpace]; ///< spare area for user data
 };
 
 // Version of this header
 // with DB marker for DataBlock start, 1st byte in header little-endian
-const uint32_t DataBlockVersion = 0x0002DBDB;
+const uint32_t DataBlockVersion = 0x0003DBDB;
 
 // DataBlockHeader instance with all default fields
-const DataBlockHeader defaultDataBlockHeader = { .headerVersion = DataBlockVersion, .headerSize = sizeof(DataBlockHeader), .dataSize = 0, .blockId = undefinedBlockId, .pipelineId = undefinedBlockId, .timeframeId = undefinedTimeframeId, .runNumber = undefinedRunNumber, .systemId = undefinedSystemId, .feeId = undefinedFeeId, .equipmentId = undefinedEquipmentId, .linkId = undefinedLinkId, .timeframeOrbitFirst = undefinedOrbit, .timeframeOrbitLast = undefinedOrbit, .flagEndOfTimeframe = 0, .isRdhFormat = 1, .userSpace = { 0 } };
+const DataBlockHeader defaultDataBlockHeader = { .headerVersion = DataBlockVersion, .headerSize = sizeof(DataBlockHeader), .dataSize = 0, .blockId = undefinedBlockId, .pipelineId = undefinedBlockId, .timeframeId = undefinedTimeframeId, .runNumber = undefinedRunNumber, .systemId = undefinedSystemId, .feeId = undefinedFeeId, .equipmentId = undefinedEquipmentId, .linkId = undefinedLinkId, .timeframeOrbitFirst = undefinedOrbit, .timeframeOrbitLast = undefinedOrbit, .flagEndOfTimeframe = 0, .isRdhFormat = 1, .orbitOffset = undefinedOrbit, .userSpace = { 0 } };
 
 // DataBlock
 // Pair of header + payload data
diff --git a/src/ReadoutEquipment.cxx b/src/ReadoutEquipment.cxx
@@ -634,8 +634,9 @@ int ReadoutEquipment::tagDatablockFromRdh(RdhHandle& h, DataBlockHeader& bh)
     isError = 1;
   } else {
     // timeframe ID
-    hbOrbit = h.getHbOrbit();
+    hbOrbit = h.getHbOrbit() + bh.orbitOffset;
     tfId = getTimeframeFromOrbit(hbOrbit);
+    // printf("orbit %X + offset %X = %X -> TFid %d\n",(int)h.getHbOrbit(), (int)bh.orbitOffset, (int)hbOrbit, (int)tfId);
 
     // system ID
     systemId = h.getSystemId();
@@ -661,6 +662,9 @@ int ReadoutEquipment::tagDatablockFromRdh(RdhHandle& h, DataBlockHeader& bh)
   bh.equipmentId = equipmentId;
   bh.linkId = linkId;
   getTimeframeOrbitRange(tfId, bh.timeframeOrbitFirst, bh.timeframeOrbitLast);
+  bh.timeframeOrbitFirst -= bh.orbitOffset;
+  bh.timeframeOrbitLast -= bh.orbitOffset;
+  // printf("TF %d eq %d link %d : orbits %X - %X\n", (int)bh.timeframeId, (int)bh.equipmentId, (int)bh.linkId, (int)bh.timeframeOrbitFirst, (int)bh.timeframeOrbitLast);
   return isError;
 }
 
diff --git a/src/ReadoutEquipmentPlayer.cxx b/src/ReadoutEquipmentPlayer.cxx
@@ -54,6 +54,7 @@ class ReadoutEquipmentPlayer : public ReadoutEquipment
   PacketHeader lastPacketHeader; // keep track of last packet header
 
   uint32_t orbitOffset = 0; // to be applied to orbit after 1st loop
+  int cfgUpdateOrbits = 1; // when set, all RDHs are modified to update orbit, according to orbitOffset
 
   void copyFileDataToPage(void* page); // fill given page with file data according to current settings
 };
@@ -97,11 +98,16 @@ ReadoutEquipmentPlayer::ReadoutEquipmentPlayer(ConfigFile& cfg, std::string cfgE
   cfg.getOptionalValue<int>(cfgEntryPoint + ".fillPage", fillPage, 1);
   // configuration parameter: | equipment-player-* | autoChunk | int | 0 | When set, the file is replayed once, and cut automatically in data pages compatible with memory bank settings and RDH information. In this mode the preLoad and fillPage options have no effect. |
   cfg.getOptionalValue<int>(cfgEntryPoint + ".autoChunk", autoChunk, 0);
-  // configuration parameter: | equipment-player-* | autoChunkLoop | int | 0 | When set, the file is replayed in loops. Trigger orbit counter in RDH are modified for iterations after the first one, so that they keep increasing. If value is negative, only that number of loop is executed (-5 -> 5x replay). |
+  // configuration parameter: | equipment-player-* | autoChunkLoop | int | 0 | When set, the file is replayed in loops. If value is negative, only that number of loop is executed (-5 -> 5x replay). |
   cfg.getOptionalValue<int>(cfgEntryPoint + ".autoChunkLoop", autoChunkLoop, 0);
+  // configuration parameter: | equipment-player-* | updateOrbits | int | 1 | When set, trigger orbit counters in all RDH are modified for iterations after the first one (in file loop replay mode), so that they keep increasing. |
+  cfg.getOptionalValue<int>(cfgEntryPoint + ".updateOrbits", cfgUpdateOrbits, 1);
 
   // log config summary
-  theLog.log(LogInfoDevel_(3002), "Equipment %s: using data source file=%s preLoad=%d fillPage=%d autoChunk=%d autoChunkLoop=%d", name.c_str(), filePath.c_str(), preLoad, fillPage, autoChunk, autoChunkLoop);
+  theLog.log(LogInfoDevel_(3002), "Equipment %s: using data source file=%s preLoad=%d fillPage=%d autoChunk=%d autoChunkLoop=%d updateOrbits=%d", name.c_str(), filePath.c_str(), preLoad, fillPage, autoChunk, autoChunkLoop, cfgUpdateOrbits);
+  if ((!cfgUpdateOrbits)&&(autoChunkLoop)) {
+    theLog.log(LogWarningDevel_(3104), "Equipment %s: RDH orbits auto-update is disabled, generated data will be inconsistent (TFid and orbit counters mismatch)", name.c_str());
+  }
 
   // open data file
   fp = fopen(filePath.c_str(), "rb");
@@ -205,6 +211,10 @@ DataBlockContainerReference ReadoutEquipmentPlayer::getNextBlock()
     // no need to fill header defaults, this is done by getNewDataBlockContainer()
     // only adjust payload size
     b->header.dataSize = 0;
+    // and possibly set orbit offset
+    if (!cfgUpdateOrbits) {
+      b->header.orbitOffset = orbitOffset;
+    }
 
     if (autoChunk) {
       bool isOk = 1;
@@ -231,6 +241,7 @@ DataBlockContainerReference ReadoutEquipmentPlayer::getNextBlock()
                 loopCount++;
                 fileOffset = 0;
                 orbitOffset = lastPacketHeader.timeframeId * getTimeframePeriodOrbits();
+		// printf("loop %d: offset = %X\n",(int)loopCount, (int)orbitOffset);
                 isOk = 1;
               }
             }
@@ -251,7 +262,7 @@ DataBlockContainerReference ReadoutEquipmentPlayer::getNextBlock()
               isOk = 0;
               break;
             }
-            if (orbitOffset) {
+            if (cfgUpdateOrbits) {
               // update RDH orbit when applicable
               h.incrementHbOrbit(orbitOffset);
             }
@@ -261,7 +272,7 @@ DataBlockContainerReference ReadoutEquipmentPlayer::getNextBlock()
             currentPacketHeader.linkId = (int)h.getLinkId();
             currentPacketHeader.equipmentId = (int)(h.getCruId() * 10 + h.getEndPointId());
 
-            int hbOrbit = h.getHbOrbit();
+            int hbOrbit = h.getHbOrbit() + b->header.orbitOffset;
             currentPacketHeader.timeframeId = getTimeframeFromOrbit(hbOrbit);
 
             // fill page metadata
diff --git a/src/ReadoutVersion.h b/src/ReadoutVersion.h
@@ -9,5 +9,5 @@
 // granted to it by virtue of its status as an Intergovernmental Organization
 // or submit itself to any jurisdiction.
 
-#define READOUT_VERSION "2.10.0"
+#define READOUT_VERSION "2.10.1"
 
diff --git a/src/readoutConfigEditor.tcl b/src/readoutConfigEditor.tcl
@@ -110,10 +110,11 @@ set configurationParametersDescriptor {
 | equipment-dummy-* | eventMinSize | bytes | 128k | Minimum size of randomly generated event. |
 | equipment-dummy-* | fillData | int | 0 | Pattern used to fill data page: (0) no pattern used, data page is left untouched, with whatever values were in memory (1) incremental byte pattern (2) incremental word pattern, with one random word out of 5. |
 | equipment-player-* | autoChunk | int | 0 | When set, the file is replayed once, and cut automatically in data pages compatible with memory bank settings and RDH information. In this mode the preLoad and fillPage options have no effect. |
-| equipment-player-* | autoChunkLoop | int | 0 | When set, the file is replayed in loops. Trigger orbit counter in RDH are modified for iterations after the first one, so that they keep increasing. If value is negative, only that number of loop is executed (-5 -> 5x replay). |
+| equipment-player-* | autoChunkLoop | int | 0 | When set, the file is replayed in loops. If value is negative, only that number of loop is executed (-5 -> 5x replay). |
 | equipment-player-* | filePath | string | | Path of file containing data to be injected in readout. |
 | equipment-player-* | fillPage | int | 1 | If 1, content of data file is copied multiple time in each data page until page is full (or almost full: on the last iteration, there is no partial copy if remaining space is smaller than full file size). If 0, data file is copied exactly once in each data page. |
 | equipment-player-* | preLoad | int | 1 | If 1, data pages preloaded with file content on startup. If 0, data is copied at runtime. |
+| equipment-player-* | updateOrbits | int | 1 | When set, trigger orbit counters in all RDH are modified for iterations after the first one (in file loop replay mode), so that they keep increasing. |
 | equipment-rorc-* | cardId | string | | ID of the board to be used. Typically, a PCI bus device id. c.f. AliceO2::roc::Parameters. |
 | equipment-rorc-* | channelNumber | int | 0 | Channel number of the board to be used. Typically 0 for CRU, or 0-5 for CRORC. c.f. AliceO2::roc::Parameters. |
 | equipment-rorc-* | cleanPageBeforeUse | int | 0 | If set, data pages are filled with zero before being given for writing by device. Slow, but usefull to readout incomplete pages (driver currently does not return correctly number of bytes written in page. |
@@ -126,9 +127,11 @@ set configurationParametersDescriptor {
 | equipment-zmq-* | type | string | SUB | Type of ZMQ socket to use to get data (PULL, SUB). |
 | readout | aggregatorSliceTimeout | double | 0 | When set, slices (groups) of pages are flushed if not updated after given timeout (otherwise closed only on beginning of next TF, or on stop). |
 | readout | aggregatorStfTimeout | double | 0 | When set, subtimeframes are buffered until timeout (otherwise, sent immediately and independently for each data source). |
+| readout | customCommands | string | | List of key=value pairs defining some custom shell commands to be executed at before/after state change commands. |
 | readout | disableAggregatorSlicing | int | 0 | When set, the aggregator slicing is disabled, data pages are passed through without grouping/slicing. |
 | readout | disableTimeframes | int | 0 | When set, all timeframe related features are disabled (this may supersede other config parameters). |
 | readout | exitTimeout | double | -1 | Time in seconds after which the program exits automatically. -1 for unlimited. |
+| readout | flushConsumerTimeout | double | 1 | Time in seconds to wait before stopping the consumers (ie wait allocated pages released). 0 means stop immediately. |
 | readout | flushEquipmentTimeout | double | 1 | Time in seconds to wait for data once the equipments are stopped. 0 means stop immediately. |
 | readout | logbookApiToken | string | | The token to be used for the logbook API. |
 | readout | logbookEnabled | int | 0 | When set, the logbook is enabled and populated with readout stats at runtime. |
@@ -144,6 +147,9 @@ set configurationParametersDescriptor {
 | readout | timeStop | string | | In standalone mode, time at which to execute stop. If not set, on int/term/quit signal. |
 | readout-monitor | broadcastHost | string | | used by readout-status to connect to readout-monitor broadcast channel. |
 | readout-monitor | broadcastPort | int | 0 | when set, the process will create a listening TCP port and broadcast statistics to connected clients. |
+| readout-monitor | logFile | string | | when set, the process will log received metrics to a file. |
+| readout-monitor | logFileHistory | int | 1 | defines the maximum number of previous log files to keep, when a maximum size is set. |
+| readout-monitor | logFileMaxSize | int | 128 | defines the maximum size of log file (in MB). When reaching this threshold, the log file is rotated. |
 | readout-monitor | monitorAddress | string | tcp://127.0.0.1:6008 | Address of the receiving ZeroMQ channel to receive readout statistics. |
 | readout-monitor | outputFormat | int | 0 | 0: default, human readable. 1: raw bytes. |
 | receiverFMQ | channelAddress | string | ipc:///tmp/pipe-readout | c.f. parameter with same name in consumer-FairMQchannel-* |
diff --git a/src/readoutErrorCodes.h b/src/readoutErrorCodes.h
@@ -33,6 +33,7 @@
   { 3101, "A feature is configured but not supported by this readout build.", nullptr},
   { 3102, "Syntax error in configuration.", nullptr},
   { 3103, "Inconsistent parameters in configuration.", nullptr},
+  { 3104, "Some valid but unsafe configuration parameters are used.", nullptr},
 
   { 3210, "Logbook problem", nullptr},
   { 3220, "Timeframe server problem", nullptr},
