Chris Dew

writes about software development

Six Integer Abuses in C

All Aplication Binary Interfaces make returning an `int` (or pointer) from a function call very efficient (placed in r0 on ARM, EAX on x86, RAX on AMD64). This has led to C developers abusing `int` when returning. As well as doing this for performance reason, C’s lack of [Algebraic Data Types](http://en.wikipedia.org/wiki/Algebraic_data_type) and Multiple Return Values means that some C developers may not fully realise what they are trying to return. I am not attempting criticize, or give alternatives to this practise, as C’s abuse of integers is idiomatic in the language. This blog post tries to shine a little light on what C developers do, for the benefit of those who know other languages. Legitimate Use ————–
1
2
3
int add(int a, int b) {
  return a + b;
}
The snippet above uses integers as they are described. Such use of integers is in the minority in real C codebases. Haskell: Int Java: int 1. As a Boolean —————
1
2
3
4
5
6
7
8
int isOk() {
  return someCondition()? 1 : 0; 
}

...
  if (isOk()) {
    ..
  }
The return value is intended to be used in an `if` statement (or tertiary operator). Haskell: Bool Java: boolean 2. As a Boolean with All Bits Set ——————————— This one is rarer, but some developers like to set all the bits for true.
1
2
3
4
5
6
7
8
int isOk() {
  return someCondition()? -1 : 0; 
}

...
  if (isOk()) {
    ..
  }
The return value is intended to be used in an `if` statement (or tertiary operator). Haskell: Bool Java: boolean 3. Success 0, Failure 1 ———————–
1
2
3
4
5
6
7
8
9
10
11
int doSomething() {
  ...
  return thingWorked() ? 0 : 1; 
}

...
  int rc = doSomething();
  if (!rc()) {
    /* handle error */
    ...
  }
This convention is exactly opposite of the two boolean conventions above. It should be thought of as ‘did it fail’? Haskell: Bool Java: boolean 3. Comparison Enumeration ————————-
1
2
3
4
5
6
7
8
int compare(int a, int b) {
  if (a > b) return 1;
  if (a < b) return -1;
  return 0;
}

...
  someSortingFunction(int *arr, int num, &compare);
This convention is exactly opposite of the two boolean conventions above. It should be thought of as ‘did it fail’? Haskell: `data Comp = LessThan | EqualTo | GreaterThan` Java: `public enum Comp { LESSTHAN, EQUALTO, GREATERTHAN };` 5. Success 0, Failure Negative Number ————————————-
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
#define SUCCESS 0
#define NOFILE -1
#define NORAM -2
#define BADARG -3

int doSomething() {
  int ret = SUCCESS;
  FILE *file;
  void *mem;

  file = openfile(...);
  if (!file) {
    ret = NOFILE;
    goto nofile;
  } 
  void *mem = malloc(123456);
  if (!mem) {
    ret = NORAM;
    goto noram;
  }
  /* do stuff */
  closefile(file);
  nofile:
  free(p);
  noram:
  return ret; 
}

...
  int rc = doSomething();
  if (!rc()) {
    /* handle error */
    ...
  }
This is a union of a (smaller) integer error code and [Unit](http://en.wikipedia.org/wiki/Unit_type). Haskell: `data Outcome = Success | Failure Int` Java: `Option` 6. Success 0, Failure Positive Number ————————————- As above, but with positive error codes. 7. Multiple Values ——————
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
#define ASEVEN 1 << 30;
#define BFIVE;

int flag(short a, short b) {
  int ret = 0;
  if (a == 7) ret |= ASEVEN; 
  if (b == 5) ret |= BFIVE; 
  return ret;
}

...
  int r = flag(a, b);
  if (r & ASEVEN) {
    /* do something */
  } else {
    /* do something else */
  }
Haskell: `data Comp = ASeven | BFive` Java:
1
2
3
4
public enum Flag { 
  ASEVEN, BFIVE; 
  public static final EnumSet<Flag> ALL_OPTS = EnumSet.allOf(Flag.class);
}

Review of Instant Vert.x

Instant Vert.x is a short introduction text for Vert.x.

There are a too many mistakes, such as

sudo echo "export PATH=\$PATH:/opt/vertx/bin/" >> /etc/profile

on page 10, which gives an error: bash: /etc/profile: Permission denied due to the redirection not being done as root (sudo only gives root permissions to the execution of the binary, not the redirection of streams).

There is then a second mistake here, as step three it makes a link /opt/vert.x, but here it uses /opt/vertx (note the mising dot).

If you know your way around Linux, you will get past these sorts of issues – but it does feel like sloppy editing.

The next error (if you’re doing the Javascript track) is on page 13.

var vertx = require(vertx);
var httpServer = vertx.createHttpServer();
httpServer.listen(8080, localhost);

Here, line 1 needs to have quotes around vertx and line 3 needs quotes around localhost to make the code work.

At this point I gave up – code samples in a beginner’s book need to be tested thoroughly.

Conclusion: I’ll have to give this book only 2/5 stars.

I hope Packt will fix these issues, as the general approach and tone of the book is good – it’s just let down by a lack of testing.

Explanation of Holographic Views of Data

As I had a few emails asking for an explanation of how ‘Holographic Views of Data’ worked, I’ve made an attempt at an explanation.

Coprime Numbers

A pair of numbers are coprime if they share no common factors (other than one).

A set of numbers is coprime if every pair of them are coprime.

Holo uses huge coprime numbers which take 32 bytes (not bits) to store.

For this example I will choose the set of coprime numbers [99, 98, 97, 95, 89].

I will produce a system where there are five views (each corresponds to a coprime) and any three can be used to recreate a an orginal number.

Original Data

Instead of ‘orginal data’, this example uses an ‘original number’. (Numbers and data are one and the same.)

This number can be between 0 and 820135 (97 * 95 * 89).

I choose the number 123456 as the ‘original number’, though any would have done.

Remainders

I then find the remainders of 123456 when divided by each of the coprimes.

These remainders are [3,74,72,51,13]. These are the five ‘views’ of the ‘original number’.

Recovery of the Original Number

I’ll choose three ‘views’ at random: 98:74, 97:72 and 89:13.

This means that the reconstruction of the number will be done in base 846034.

We’ll need a coefficient for each of the coprimes. We calculate these as:

gmpy.invert((97 * 89) % 98, 98) = 11 gmpy.invert((98 * 89) % 97, 97) = 12 gmpy.invert((98 * 97) % 89, 89) = 68

We can then recreate the original number by multiplying each view by the appropriate coefficient and summing them, mod 846034.

(74 * 11 + 72 * 12 + 13 * 68) % 846034 = doesn’t work yet

Workings

Prelude> map (gcd 98) [99] [1] Prelude> map (gcd 97) [99, 98] [1,1] Prelude> map (gcd 96) [99, 98, 97] [3,2,1] Prelude> map (gcd 95) [99, 98, 97] [1,1,1] Prelude> map (gcd 94) [99, 98, 97, 95] [1,2,1,1] Prelude> map (gcd 93) [99, 98, 97, 95] [3,1,1,1] Prelude> map (gcd 92) [99, 98, 97, 95] [1,2,1,1] Prelude> map (gcd 91) [99, 98, 97, 95] [1,7,1,1] Prelude> map (gcd 90) [99, 98, 97, 95] [9,2,1,5] Prelude> map (gcd 89) [99, 98, 97, 95] [1,1,1,1]

Prelude> map (mod 123456) [99, 98, 97, 95, 89] [3,74,72,51,13]

Reboot on Kernel Panic

Last week, a server crashed with a kernel panic. It stayed crashed all night.

This seems to be the default behaviour wuth Ubuntu server 12.04.

I did some research. Running the following commands will make a machine reboot 15 seconds after a kernel panic.

# fix the issue now
echo "15" | sudo tee /proc/sys/kernel/panic
# fix the issue on subsequent reboots
echo "kernel.panic = 15" | sudo tee /etc/sysctl.d/10-reboot-on-panic.conf

I’ve now done this for all my servers.

Kernel panics shouldn’t happen. But if they do, I want a speedy reboot, not a screen full of CPU registers sitting there all night long.

Holographic Views of Data

An algorithm for creating unlimited, redundant, orthogonal views of data – and regenerating original data from any available views (whose aggregate size exceed that of the original data).

Update

Thanks to everyone for your comments by email – chris@chrisdew.com

I can summarise: I am completely out of my depth here – there is extensive work already done on this area:

I had not encountered any obvious explicit uses of this type of work, though it may be in use at lower levels of communication protocols.

Where is my HyperZFS filesystem? (Where my data is accessible so long as I have working disks whose aggregate size exceeds the size of my data.)

TL;DR;

I’ve just invented a fountain code. As far as I can tell, it’s not the LT Code, the Raptor Code or an Online Code. (I don’t use XOR or polynomials, which LT and Raptor do, and it’s a single phase algorithm, which Online Codes aren’t).

I don’t know whether what I’ve invented is faster than the existing fountain codes, at ~100MB/s/core on an i5. The single core en/decoding rate is similar to the rate at which you can read from spinning rust. You need several cores to saturate an SSD.

If you have any interest in this work, please contact me: chris@chrisdew.com.

Transcript

Hi, I’d like to ask whether an algorithm that I’ve developed, is new or useful.

I’m a Software Developer by trade but my background is in physics, not computer science.

Therefore I may have just reinvented the wheel here and have been too ignorant to see it.

The algorithm is inspired by holography, where every part of a hologram contains some of the information about all of the scene – I’ve named the binary ‘holo’.

Anyway, on to the demonstration…

First I’ll need some chunky data, to demonstrate that the algorithm isn’t stupidly slow.

For this I’ll use an Ubuntu iso image, though obviously the algorithm works on any type of binary data.

I’ll give the iso image a shorter name to save typing.

chris@tux.io:~/screencast$ cp ~/Downloads/ubuntu-12.04.2-desktop-amd64.iso iso
chris@tux.io:~/screencast$ ll
total 711688
drwxrwxr-x  2 chris chris      4096 Aug 15 21:36 ./
drwxr-xr-x 42 chris chris      4096 Aug 15 21:09 ../
-rw-r--r--  1 chris chris 728760320 Aug 15 21:36 iso

700MB of test data.

Let’s get a checksum of the file, so that we can guard against any file corruption.

chris@tux.io:~/screencast$ sha1sum iso
f94d8591dad3043634a35e23884306f16b6b5fc3  iso
chris@tux.io:~/screencast$ time holo --generate --size=1/4 --views=8 iso

I’ve asked for eight views, each a quarter the size of the original, but you could ask for any amount of any size.

This will take a moment, as we’re dealing with a few hundred megs of data.

generating views from iso...

real    0m12.594s
user    0m11.749s
sys     0m0.660s

That’s finished.

If I list the directory we can see the original iso, and the eight generated views.

chris@tux.io:~/screencast$ ll
total 2146316
drwxrwxr-x  2 chris chris      4096 Aug 15 21:37 ./
drwxr-xr-x 42 chris chris      4096 Aug 15 21:09 ../
-rw-r--r--  1 chris chris 728760320 Aug 15 21:36 iso
-rw-rw-r--  1 chris chris 183624680 Aug 15 21:37 iso.view0
-rw-rw-r--  1 chris chris 183624680 Aug 15 21:37 iso.view1
-rw-rw-r--  1 chris chris 183624680 Aug 15 21:37 iso.view2
-rw-rw-r--  1 chris chris 183624680 Aug 15 21:37 iso.view3
-rw-rw-r--  1 chris chris 183624680 Aug 15 21:37 iso.view4
-rw-rw-r--  1 chris chris 183624680 Aug 15 21:37 iso.view5
-rw-rw-r--  1 chris chris 183624680 Aug 15 21:37 iso.view6
-rw-rw-r--  1 chris chris 183624680 Aug 15 21:37 iso.view7

Let’s do some maths, to check the sizes.

chris@tux.io:~/screencast$ ghci
GHCi, version 7.4.1: http://www.haskell.org/ghc/  :? for help
Loading package ghc-prim ... linking ... done.
Loading package integer-gmp ... linking ... done.
Loading package base ... linking ... done.
Prelude> 183624680 * 4.0 / 728760320
1.0078741938090152
Prelude>
Leaving GHCi.

We can see that four pieces together are less than one percent larger than the original, so this is pretty space efficient.

Let’s delete the original file.

chris@tux.io:~/screencast$ rm iso
chris@tux.io:~/screencast$ ll
total 1434632
drwxrwxr-x  2 chris chris      4096 Aug 15 21:38 ./
drwxr-xr-x 42 chris chris      4096 Aug 15 21:09 ../
-rw-rw-r--  1 chris chris 183624680 Aug 15 21:37 iso.view0
-rw-rw-r--  1 chris chris 183624680 Aug 15 21:37 iso.view1
-rw-rw-r--  1 chris chris 183624680 Aug 15 21:37 iso.view2
-rw-rw-r--  1 chris chris 183624680 Aug 15 21:37 iso.view3
-rw-rw-r--  1 chris chris 183624680 Aug 15 21:37 iso.view4
-rw-rw-r--  1 chris chris 183624680 Aug 15 21:37 iso.view5
-rw-rw-r--  1 chris chris 183624680 Aug 15 21:37 iso.view6
-rw-rw-r--  1 chris chris 183624680 Aug 15 21:37 iso.view7

Now for some magic. From any four pieces we can recreate the original file.

chris@tux.io:~/screencast$ time holo --regenerate iso.view0 iso.view7 iso.view3 iso.view5
regenerating iso from iso.view0 iso.view7 iso.view3 iso.view5...

The requirements for recreating the original are just that the total size of the views used are larger than the size of the original by at least one bit.

There is nothing in the algorithm preventing the parts from being of different sizes, it was just easier for this implementation to make them all the same.

The algorithm itself is embarrassingly parallel, but my implementation is unoptimised code running single threaded on a CPU.

A GPU implementation may be much faster.

real    0m8.631s
user    0m8.157s
sys     0m0.460s

That’s finished, lets check it’s worked.

chris@tux.io:~/screencast$ ll
total 2146312
drwxrwxr-x  2 chris chris      4096 Aug 15 21:38 ./
drwxr-xr-x 42 chris chris      4096 Aug 15 21:09 ../
-rw-rw-r--  1 chris chris 728760320 Aug 15 21:38 iso
-rw-rw-r--  1 chris chris 183624680 Aug 15 21:37 iso.view0
-rw-rw-r--  1 chris chris 183624680 Aug 15 21:37 iso.view1
-rw-rw-r--  1 chris chris 183624680 Aug 15 21:37 iso.view2
-rw-rw-r--  1 chris chris 183624680 Aug 15 21:37 iso.view3
-rw-rw-r--  1 chris chris 183624680 Aug 15 21:37 iso.view4
-rw-rw-r--  1 chris chris 183624680 Aug 15 21:37 iso.view5
-rw-rw-r--  1 chris chris 183624680 Aug 15 21:37 iso.view6
-rw-rw-r--  1 chris chris 183624680 Aug 15 21:37 iso.view7

And its checksum.

chris@tux.io:~/screencast$ sha1sum iso
f94d8591dad3043634a35e23884306f16b6b5fc3  iso

Let’s repeat that, choosing a different set of parts… any selection of four different parts will work.

chris@tux.io:~/screencast$ rm iso
chris@tux.io:~/screencast$ ll
total 1434632
drwxrwxr-x  2 chris chris      4096 Aug 15 21:39 ./
drwxr-xr-x 42 chris chris      4096 Aug 15 21:09 ../
-rw-rw-r--  1 chris chris 183624680 Aug 15 21:37 iso.view0
-rw-rw-r--  1 chris chris 183624680 Aug 15 21:37 iso.view1
-rw-rw-r--  1 chris chris 183624680 Aug 15 21:37 iso.view2
-rw-rw-r--  1 chris chris 183624680 Aug 15 21:37 iso.view3
-rw-rw-r--  1 chris chris 183624680 Aug 15 21:37 iso.view4
-rw-rw-r--  1 chris chris 183624680 Aug 15 21:37 iso.view5
-rw-rw-r--  1 chris chris 183624680 Aug 15 21:37 iso.view6
-rw-rw-r--  1 chris chris 183624680 Aug 15 21:37 iso.view7
chris@tux.io:~/screencast$ time holo --regenerate iso.view1 iso.view2 iso.view3 iso.view6
regenerating iso from iso.view1 iso.view2 iso.view3 iso.view6...

real    0m8.743s
user    0m8.165s
sys     0m0.536s
chris@tux.io:~/screencast$ ll
total 2146312
drwxrwxr-x  2 chris chris      4096 Aug 15 21:39 ./
drwxr-xr-x 42 chris chris      4096 Aug 15 21:09 ../
-rw-rw-r--  1 chris chris 728760320 Aug 15 21:39 iso
-rw-rw-r--  1 chris chris 183624680 Aug 15 21:37 iso.view0
-rw-rw-r--  1 chris chris 183624680 Aug 15 21:37 iso.view1
-rw-rw-r--  1 chris chris 183624680 Aug 15 21:37 iso.view2
-rw-rw-r--  1 chris chris 183624680 Aug 15 21:37 iso.view3
-rw-rw-r--  1 chris chris 183624680 Aug 15 21:37 iso.view4
-rw-rw-r--  1 chris chris 183624680 Aug 15 21:37 iso.view5
-rw-rw-r--  1 chris chris 183624680 Aug 15 21:37 iso.view6
-rw-rw-r--  1 chris chris 183624680 Aug 15 21:37 iso.view7
chris@tux.io:~/screencast$ sha1sum iso
f94d8591dad3043634a35e23884306f16b6b5fc3  iso

The same original file, created from a different set of views.

I’ve tried to think of some practical uses for this, but I’d love to hear your ideas, on Hacker News, or Reddit.

Thanks for taking the time to watch this. Please let me know if this algorithm already exists and if so, a wikipedia link would be great.

Beware android.R

I have just wasted more than an hour of my life. My Android Studio project just stopped compiling, coming up with errors about symbols in R not being found.

I persued my usual IDE’s-messed-itself-up solution of moving the project aside, creating a new one of the same name, then using meld to make the new one like the old one, one step at a time.

The problem was that Android Studio had helpfully inserted an innocuous-looking import android.R line.

R normally exists in your app’s package, so this hid all of the resources.

Android Studio NoSuchMethodError LazyStringArrayList

While evaluating Android Studo an error was stopping me from building my app. I found a workaround and habe blogged it in case anyone else has the same difficulty.

System

Android Studio (I/O Preview) AI-130.737825
Build #AI-130.737825, built on July 11, 2013
JRE : 1.6.0_45
VM: Java HotSpot(TM) 64-Bit Server VM by Sun Microsystems Inc.

64 bit Ubuntu 12.04LTS

Android Studio installed in /opt

Error

NoSuchMethodError: com.google.protobuf.LazyStringArrayList.(Lcom/google/protobuf/LazyStringList;)V: com.google.protobuf.LazyStringArrayList.(Lcom/google/protobuf/LazyStringList;)V

[  27409]  ERROR - ij.compiler.impl.CompileDriver - com.google.protobuf.LazyStringArrayList.<init>(Lcom/google/protobuf/LazyStringList;)V 
java.lang.NoSuchMethodError: com.google.protobuf.LazyStringArrayList.<init>(Lcom/google/protobuf/LazyStringList;)V
at org.jetbrains.jps.api.CmdlineRemoteProto$Message$ControllerMessage$ParametersMessage$TargetTypeBuildScope$Builder.ensureTargetIdIsMutable(CmdlineRemoteProto.java:4195)
at org.jetbrains.jps.api.CmdlineRemoteProto$Message$ControllerMessage$ParametersMessage$TargetTypeBuildScope$Builder.addAllTargetId(CmdlineRemoteProto.java:4256)
at org.jetbrains.jps.api.CmdlineProtoUtil.createTargetsScope(CmdlineProtoUtil.java:74)
at com.android.tools.idea.gradle.compiler.AndroidGradleBuildTargetScopeProvider.getBuildTargetScopes(AndroidGradleBuildTargetScopeProvider.java:46)
at com.intellij.compiler.impl.CompileDriver.compileInExternalProcess(CompileDriver.java:495)
at com.intellij.compiler.impl.CompileDriver.access$200(CompileDriver.java:118)
at com.intellij.compiler.impl.CompileDriver$8.run(CompileDriver.java:703)
at com.intellij.compiler.progress.CompilerTask.run(CompilerTask.java:167)
at com.intellij.openapi.progress.impl.ProgressManagerImpl$TaskRunnable.run(ProgressManagerImpl.java:502)
at com.intellij.openapi.progress.impl.ProgressManagerImpl$2.run(ProgressManagerImpl.java:185)
at com.intellij.openapi.progress.impl.ProgressManagerImpl.executeProcessUnderProgress(ProgressManagerImpl.java:229)
at com.intellij.openapi.progress.impl.ProgressManagerImpl.runProcess(ProgressManagerImpl.java:175)
at com.intellij.openapi.progress.impl.ProgressManagerImpl$8.run(ProgressManagerImpl.java:404)
at com.intellij.openapi.application.impl.ApplicationImpl$8.run(ApplicationImpl.java:431)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)
at com.intellij.openapi.application.impl.ApplicationImpl$1$1.run(ApplicationImpl.java:155)

Issue

I’m not sure what the issue is, other than that there are two versions of protobuf in the lib directory.

Work Around

chris@chris-thorcom:~$ cd /opt/android-studio/lib
chris@chris-thorcom:/opt/android-studio/lib$ mv protobuf-2.4.1.jar protobuf-2.4.1.jar.disabled

Tenacious vs Stubborn

Success

You’ve believed in your vision, despite the being told it’ll never work.

You’ve remortgaged your house and quit your job to spend 5 years on your project.

Five years later you’ve succeeded due to your tenacity and vision.

Failure

You’ve believed in your vision, despite the being told it’ll never work.

You’ve remortgaged your house and quit your job to spend 4 years on your project.

Four years later you’ve failed due to your stubborness and delusions.

Difference

Can you tell (without hindsight) whether you are being tenacious and visionary, rather than stubborn and deluded?

Dates Are Time Intervals

Recently, while writing some time-sensitive reporting software, I had a minor insight…

To me (and I may or may not be representative of the human population):

2013-06-20T13:02:22.132 isn’t a millisecond, it is the start of that millisecond.

2013-06-20T13:02:23 isn’t a minute, it is the start of that minute.

2013-06-20 is a day, the interval [2013-06-20T00:00:00, 2013-06-21T00:00:00)

(Google “half-open interval” if the above brackets are not familiar.)

A UI which has a start and a finish datepicker is allowing the user to pick two time intervals which implicitly specify a larger time interval from the beginning of start to the end of finish.

(English’s redundancy (e.g. start/begin and finish/end) is a great boon for programmers, whose main task in life is to name things. According to the bible, this was humanity’s original purpose – Genesis 2:19.)

Fundamentaly there is ambiguity about whether each Date/Time value in an app is implicitly an interval. It’s easy to decide that Date/Times are all just points in time, but that’s not how people read UI widgets displaying them.

Potentially all software needs to know (ideally at the type level) whether each Date/Time is an implicit (day long? month long?) interval, or a point in time. Do any languages/databases do this?

For my app, the upshot was just to add 86400000ms to all finish fields when getting and subtract the same before setting.

JSON Stream

The standard for delimiting JSON in stream protocols (such as TCP).

1. Introduction

This is a minimal specification for sending and receiving JSON objects over a stream protocol, such as TCP.

The JSON Stream framing is so simple that no specification had previously been written for what (to us) seemed the ‘obvious’ way to do it.

1.1 Motivation

There is currently no standard for transporting JSON within a stream protocol (primarily plain TCP), apart from Websockets, which is unneccesarily complex for non-browser applications.

There were numerous possibilities for JSON framing, including counted strings and non-ASCII delimiters (DLE STX ETX or Websocket’s 0xFFs).

1.2 Scope

The primary use case for JSON Stream delimiting is an unending stream of JSON objects, delivered at variable times, over TCP, where each object needs to be processed as it arrives. e.g. a stream of stock quotes or chat messages.

2. Philosphy / Requirements

The specification must be trivial to implement in multiple programming languages, flexible enough to handle arbitrary whitespace (pretty-printed JSON), and yet not contain non-printable characters.

3. Functional Specification

3.1 Sending

Each JSON object MUST be written to the stream followed by the new line character 0x0A. The JSON objects MAY contain newlines, carriage returns and any other permitted whitespace. See www.json.org for the full spec.

All serialized data MUST use the UTF8 encoding.

3.2 Receiving

The receiver MUST accumulate received lines. Every time a line ending is encoutered, it must attempt to parse the accumulated lines into a JSON object.

The receiver MUST accept all common line endings: ‘0x0A’ (Unix), ‘0x0D’ (Mac), ‘0x0D0xA’ (Windows).

If the parsing of the accumulated lines is successful, the accumulated lines MUST be discarded and the the parsed object given to the application code.

If the amount of unparsed, accumulated characters exceeds 16MiB the receiver MAY close the the stream. Resourse constrained devices MAY close the stream at a lower threshold, though they MUST accept at least 1KiB.

The reference NodeJS/Javascript implementation will be posted on github shortly.