Skip to content

Toro Cloud Dev Center


Gloop model array icon

Gloop cursors

Occasionally when writing software you need to deal with millions of rows of data, not thousands. When dealing with this much data, it's very easy to run out of memory and/or reduce the performance of your code. To get around this, software sometimes use the concept of a cursor, or iterator, which is a pointer to a single record in a list of data. This way all the data isn't stored in memory. Instead, as the data is processed, old records in memory are replaced with new records. Gloop is no different to this. Gloop cursors will appear as an array in Martini so you can still use it the same way as any other Gloop object array, but it's important to know that Gloop cursors come with some limitations:

  • They can only be iterated over once;
  • Some cursors can only be read from; and
  • Some cursors can only be written to.

Currently, Gloop uses cursors under the following conditions only:

  • Models returned from Gloop batch SQL services are cursors that can only be written to;
  • Models returned from Gloop multi-select SQL services are cursors that can only be read from;
  • Models returned from flat file reading and writing services;
  • Reading large plain-text files via FileMethods.getTextFileInputCursor(...);
  • Reading large XML files via XmlMethods.getInputCursorFromFile(...) or XmlMethods.getInputCursorFromInputStream(...);
  • Reading large JSON files via JsonMethods.getInputCursorFromFile(...) or JsonMethods.getInputCursorFromInputStream(...); or
  • Writing large JSON files via JsonMethods.openJsonOutputCursor(...).

Below is a screenshot showing a service that returns a Gloop cursor, but still shows as an array in Martini:

Gloop cursor output

Gloop cursor output

When cursors are used as the input and outputs of iterate and while steps, they will automatically read and write from the cursors for you. However, if you need to use the cursors manually, you can use the Gloop cursor-based methods in the io.toro.martini.GloopMethods class (which is in the core package). This class includes methods for reading, writing, and closing Gloop cursors.

Examples

There are services in the examples package that demonstrate how to use Gloop cursors. These are:

  • services in the sqlServices.invoke.multiple package;
  • flatfile.ReadFile;
  • flatfile.WriteFile;
  • jsonCursor.input.ReadAndWriteSimpleJson;
  • jsonCursor.output.WriteComplexJson; and
  • xmlCursor.ReadLargeXmlFile.