REST API

This is the API reference documentation for the Triton Manta Storage Service, which enables you to store data in the cloud and process that data using the built-in compute facility.

This document covers only the HTTP interface and all examples are given in curl.

Before you do the examples in this section, it's important to go through the examples using the CLI and setting up your environment.

There are also detailed reference materials:

Conventions

Any content formatted like this:

curl -is https://us-central.manta.mnx.io

is a command-line example that you can run from a shell. All other examples and information are formatted like this:

GET /my/stor/foo HTTP/1.1

Authentication

There are a few access methodologies. The predominant means of authenticating requests to the service is to use the HTTP Signature over TLS.

In most situations, you will only need to sign the lowercase date: and value of the HTTP Date header using your SSH private key; doing this allows you to create interactive shell functions (see below). All requests require an HTTP Authorization header where the scheme is Signature.

Full details are available in the http signatures specification, but a simple form is:

Authorization: Signature keyId="/:login/keys/:fp",algorithm="rsa-sha256",signature="$base64_signature"

The keyId for the service is always /$your_triton_login/keys/$ssh_fingerprint, and the supported algorithms are: rsa-sha1, rsa-sha256 and dsa-sha. The ssh key fingerprint must be a MD5 fingerprint (ex. a1:b2:c3:d4:e5:f6:a7:b8:c9:d0:e1:f2:a3:b4:c5:d6)

To make a request for an RBAC subuser, change the keyId for the signature to /$your_triton_login/$subuser_login/keys/$ssh_fingerprint. To make a request using a RBAC role, include the HTTP header Role.

Signed URLS

It can be desirable to share "expiring links" with someone via a simple link, suitable for a web browser or email client. In these cases, the service supports a "signed URL" scheme, as an alternative to making programmatic REST requests (there is no Authorization header). Signed URLs are the only exception to the HTTPS-only requirement of the service; you are allowed to perform GET requests only over HTTP via this mechanism to avoid "mixed content" errors in some web browsers. The query string must at minimum must contain the same identifying parameters that would be in the Authorization header (user, keyId, algorithm). Additionally it must contain an expires parameter, which is a time the URL is valid until, specified as the number of seconds from the Unix epoch.

To construct a signed URL, you:

Formalized:

HTTP REQUEST Method + '\n'
Host Header + '\n'
HTTP REQUEST URI + '\n
key=val&key=val... (query parameters, url, sorted lexicographically)

For example, suppose you wanted to share a link to https://us-central.manta.mnx.io/$MANTA_USER/stor/image.png. The following would be the signing string (newlines inserted for readability):

GET\n
us-central.manta.mnx.io\n
/$MANTA_USER/stor/image.png
algorithm=RSA-SHA256&expires=1354201912&keyId=%2F$MANTA_USER%2Fkeys%2F8e%3A36%3A43%3Aac%3Ad0%3A61%3A60%3A18%3A20%3Af5%3Ab7%3Aec%3A3a%3Ad8%3A79%3A2d

You would sign that entire string with your private key, URL encode the signature and then append it to the URL query string:

https://us-central.manta.mnx.io/$MANTA_USER/stor/image.png?algorithm=RSA-SHA256&expires=1354201912&keyId=%2F$MANTA_USER%2Fkeys%2F8e%3A36%3A43%3Aac%3Ad0%3A61%3A60%3A18%3A20%3Af5%3Ab7%3Aec%3A3a%3Ad8%3A79%3A2d&signature=RR5s5%2Fa0xpwvukU2tn3LAe2QRHGRJVdbWu%2FQZk%2BYgnVmuWI59n8EG0G6KN5INp30r7xC0EOSMvgmyfrLFQTG1482fNsjedwfFXVZq0%2BeUV6dI36pe69FxMuRKh4ILy47l6wCqD4qvFsFwmeqmzfkn03MmU15JsJt2yqTtnz%2FGkToZCpaHugW5YferGNeAY%2FrTwLTrB%2BrsKovY35rK9eokPbJTDlNx97JX5%2F7ol3cgtRbstLuROfpCycJ5OxC3NAeXeUD7weGQxAY6ypoEq5HFiZoA3gT4lDdYyO7LKKPkE8dSqcqVqgdtflpf%2FYibKwGg5Vm%2F9Ze%2Fwq%2Bsb1RgSAJsA%3D%3D

Interacting with the Triton Manta Storage Service from the shell (bash)

Most things in the service are easy to interact with via cURL, but note that all requests to the service must be authenticated. You can string together OpenSSL and cURL with a bash function.

Copy all of below:

  function manta {
        local alg=rsa-sha256
        local keyId=/$MANTA_USER/keys/$MANTA_KEY_ID
        local now=$(date -u "+%a, %d %h %Y %H:%M:%S GMT")
        local sig=$(echo "date:" $now | \
                    tr -d '\n' | \
                    openssl dgst -sha256 -sign $HOME/.ssh/id_rsa | \
                    openssl enc -e -a | tr -d '\n')

        curl -sS $MANTA_URL"$@" -H "date: $now"  \
            -H "Authorization: Signature keyId=\"$keyId\",algorithm=\"$alg\",signature=\"$sig\""
    }

Paste into ~/.bash_profile or ~/.bashrc and restart your terminal to pick up the changes.

pbpaste > ~/.bash_profile

And edit the file, replacing $TRITON_CLOUD_USER_NAME with your actual cloud username.

This all is setup correctly you will be able to:

manta /$MANTA_USER/stor
$ manta /$MANTA_USER/stor/moved -X PUT -H "content-type: application/json; type=file"
$ manta /$MANTA_USER/stor/foo -X PUT -H "content-type: application/json; type=directory"
$ manta /$MANTA_USER/stor -X GET
{"name":"foo","type":"directory","mtime":"2013-06-16T05:42:56.515Z"}
  {"name":"moved","etag":"bfaa3227-3abb-4ed6-915a-a2179f623172","size":0,"type":"object","mtime":"2013-06-16T05:42:45.460Z"}

All sample "curl" requests in the rest of this document use the function above. Throughout the rest of the document, the value of the Authorization header is simply represented as $Authorization.

Errors

All HTTP requests can return user or server errors (HTTP status codes >= 400). In these cases, you can usually expect a JSON body to come along that has the following structure:

{
  "code": "ProgrammaticCode",
  "message: "human consumable message"
}

The complete list of codes that will be sent are:

Additionally, jobs may emit the above errors, or:

Error nameReason
TaskInitErrorFailed to initialize a task (usually a failure to load assets).
UserTaskErrorUser's script returned a non-zero status or one of its processes dumped core.

Directories

PutDirectory (PUT /:login/stor/[:directory]/:directory)

PutDirectory in the Triton Manta Storage Service is an idempotent create-or-update operation. Your private namespace starts at /:login/stor, and you can then create any nested set of directories or objects underneath that. To put a directory, simply set the HTTP Request-URI to the path you want to update, and set the Content-Type HTTP header to application/json; type=directory. There is no request or response body. An HTTP status code of 204 is returned on success.

Sample Request

manta /$MANTA_USER/stor/foo \
    -X PUT \
    -H 'content-type: application/json; type=directory'

PUT /$MANTA_USER/stor/foo HTTP/1.1
Host: us-central.manta.mnx.io
Accept: */*
content-type: application/json; type=directory
Date: Thu, 29 Nov 2012 23:48:00 GMT
Authorization: $Authorization

HTTP/1.1 204 No Content
Date: Thu, 29 Nov 2012 23:48:00 GMT
Server: Manta
x-request-id: 3591a050-3a7f-11e2-b95c-a921ce711752
x-response-time: 16
x-server-name: 00aa9214-0855-474d-92b5-8f713495b8d7
Connection: keep-alive

ListDirectory (GET /:login/stor/:directory)

Lists the contents of a directory. On success you will receive a \n separated stream of JSON objects, where each object represents a single directory entry. The content-type will be application/x-json-stream; type=directory. Each object will have a type field, which indicates whether the entry is a directory or an object. For example (additional newlines added for clarity):

{
    "name": "1c1bf695-230d-490e-aec7-3b11dff8ef32",
    "type": "directory",
    "mtime": "2012-09-11T20:28:30Z"
}

{
    "name": "695d5de6-45f4-4156-b6b7-3a8d4af89391",
    "etag": "bdf0aa96e3bb87148be084252a059736",
    "size": 44,
    "type": "object",
    "mtime": "2012-09-11T20:28:31Z"
}

The type field indicates the "schema" for each record; the only types are currently object and directory. Both have a name (filename), type (already described) and an mtime, which is an ISO8601 timestamp of the last update time. Additionally, records of type object have a size (content-length) and etag (for conditional requests).

You will get back entries in blocks of 256 (you can opt for less, or more by setting the limit parameter on the query string). You can choose where to start the next listing by using the marker query parameter. You'll get the total number of records in the result-set-size header. The service lists objects in alphabetical order (UTF-8 collation).

Query Parameters

NameDescription
limitlimits the number of records to come back (default and max is 1000)
markerkey name at which to start the next listing

Returns

A stream of JSON objects, one record for each child.

manta /$MANTA_USER/stor/

GET /$MANTA_USER/stor HTTP/1.1
Host: us-central.manta.mnx.io
Accept: */*
Authorization: $Authorization

HTTP/1.1 200 OK
Content-Type: application/x-json-stream; type=directory
Result-Set-Size: 1
Date: Fri, 30 Nov 2012 00:24:28 GMT
Server: Manta
x-request-id: 4db4ba00-3a84-11e2-b95c-a921ce711752
x-response-time: 8
x-server-name: 00aa9214-0855-474d-92b5-8f713495b8d7
Connection: keep-alive
Transfer-Encoding: chunked

{"name":"foo","type":"directory","mtime":"2012-11-29T23:48:00Z"}

DeleteDirectory (DELETE /:login/stor/:directory)

Deletes a directory. The directory must be empty. There is no response data from this request. On success an HTTP 204 is returned;

Sample Request

manta /$MANTA_USER/stor/foo -X DELETE

DELETE /$MANTA_USER/stor/foo HTTP/1.1
Host: us-central.manta.mnx.io
Accept: */*
Date: Fri, 30 Nov 2012 00:31:00 GMT
Authorization: $Authorization

HTTP/1.1 204 No Content
Last-Modified: Thu, 29 Nov 2012 23:48:00 GMT
Date: Fri, 30 Nov 2012 00:31:00 GMT
Server: Manta
x-request-id: 371e7320-3a85-11e2-8d0c-417db534d10b
x-response-time: 15
x-server-name: fb07e9ec-5137-418e-aff2-01d00aff1a49
Connection: keep-alive

Objects

PutObject (PUT /:login/stor/[:directory]/:object)

Creates or overwrites an object. You specify the path to an object just as you would on a traditional file system, and the parent must be a directory. The service will do no interpretation of your data. Specifically, that means your data is treated as an opaque byte stream, and you will receive back exactly what you upload. On success an HTTP 204 is returned.

By default, The service will store two copies of your data on two physical servers in two different datacenters; note that each physical server is configured with RAID-Z, so a disk drive failure does not impact your durability or availability. You can increase (or decrease) the number of copies of your object with the durability-level header.

You should always specify a Content-Type header, which will be stored and returned back (HTTP content-negotiation will be handled). If you do not specify one, the default is application/octet-stream.

You should specify a Content-MD5 header; if you do the service will validate that the content uploaded matches the value of the header. Even if you do not include one, successful responses will include a computed-md5 header. This is the MD5 checksum that Manta calculated when ingesting your object.

The service is able to provide test/set semantics for you if you use HTTP conditional request semantics (e.g., If-Match or If-Modified-Since).

Cross-Origin Resource Sharing CORS headers are saved as appropriate; preflighted requests are supported by sending a list of values in access-control-allow-origin.

By default, the service will store 2 copies of your object; this can be changed with the durability-level header.

Note that if you are using "streaming uploads" (transfer-encoding chunked), you are either subject to the default object size in the service (5Gb), or you need to "guess" how big your object is going to be by overriding with the HTTP header max-content-length. Either way, the service will track how many bytes were _actually_ sent and record that. Subsequent GET requests will return the actual length. If you're not streaming, just set content-length (as you normally would).

Lastly, you can store custom headers with your object (e.g. "tags") by prefixing them with m-. So for example, including the header m-local-user: foo with an object will allow you to track that some local user to your application created the object. You are allowed up to 4 KB of header data.

Sample Request

manta /$MANTA_USER/stor/foo.json -X PUT -H 'content-type: application/json' \
        -d '{"hello": "world"}'

PUT /$MANTA_USER/stor/foo.json HTTP/1.1
Host: us-central.manta.mnx.io
Accept: */*
content-type: application/json
Date: Fri, 30 Nov 2012 00:55:06 GMT
Authorization: $Authorization
Content-Length: 18

{"hello": "world"}


HTTP/1.1 204 No Content
Etag: f501ffd1-3e28-49a8-aaa5-2c1555c34ce0
Last-Modified: Fri, 30 Nov 2012 00:55:06 GMT
Date: Fri, 30 Nov 2012 00:55:06 GMT
Server: Manta
x-request-id: 952ddbb0-3a88-11e2-b95c-a921ce711752
x-response-time: 18
x-server-name: 00aa9214-0855-474d-92b5-8f713495b8d7
Connection: keep-alive

PutMetadata (PUT /:login/stor/[:directory]/:object?metadata=true)

PutMetadata allows you to overwrite the HTTP headers for an already existing object, without changing the data. Note this is an idempotent "replace" operation, so you must specify the complete set of HTTP headers you want stored on each request.

You cannot change "critical" headers:

Sample Request

manta /$MANTA_USER/stor/foo.json?metadata=true -X PUT \
        -H 'content-type: application/json'

PUT /$MANTA_USER/stor/foo.json?metadata=true HTTP/1.1
Host: us-central.manta.mnx.io
Accept: */*
content-type: application/json
m-foo: bar
Date: Fri, 30 Nov 2012 00:55:06 GMT
Authorization: $Authorization


HTTP/1.1 204 No Content
Etag: f501ffd1-3e28-49a8-aaa5-2c1555c34ce0
Last-Modified: Fri, 30 Nov 2012 00:55:06 GMT
Date: Fri, 30 Nov 2012 00:55:06 GMT
Server: Manta
x-request-id: caf3ccbb-5138-43c1-9c27-2bf8f13f76f3
x-response-time: 18
x-server-name: 00aa9214-0855-474d-92b5-8f713495b8d7
Connection: keep-alive

GetObject (GET /:login/stor/[:directory]/:object)

Retrieves an object from the service. On success an HTTP 200 is returned along with your content and metadata (HTTP headers).

Sample Request

manta /$MANTA_USER/stor/foo.json

GET /$MANTA_USER/stor/foo.json HTTP/1.1
Host: us-central.manta.mnx.io
Accept: */*
Date: Fri, 30 Nov 2012 00:59:17 GMT
Authorization: $Authorization


HTTP/1.1 200 OK
Etag: f501ffd1-3e28-49a8-aaa5-2c1555c34ce0
Last-Modified: Fri, 30 Nov 2012 00:55:06 GMT
Content-Length: 18
Content-Type: application/json
Content-MD5: Sd/dVLAcvNLSq16eXua5uQ==
Date: Fri, 30 Nov 2012 00:59:18 GMT
Server: Manta
x-request-id: 2af906b0-3a89-11e2-8d0c-417db534d10b
x-response-time: 7
x-server-name: fb07e9ec-5137-418e-aff2-01d00aff1a49
Connection: keep-alive

{"hello": "world"}

DeleteObject (DELETE /:login/stor/[:directory]/:object)

Deletes an object from the service. On success an HTTP 204 is returned.

Sample Request

manta /$MANTA_USER/stor/foo.json -X DELETE

DELETE /$MANTA_USER/stor/foo/bar.json HTTP/1.1
User-Agent: curl/7.21.2 (i386-pc-solaris2.11) libcurl/7.21.2 OpenSSL/0.9.8w zlib/1.2.3
Host: us-central.manta.mnx.io
Accept: */*
Date: Fri, 30 Nov 2012 01:01:33 GMT
Authorization: $Authorization


HTTP/1.1 204 No Content
Etag: f501ffd1-3e28-49a8-aaa5-2c1555c34ce0
Last-Modified: Fri, 30 Nov 2012 00:55:06 GMT
Date: Fri, 30 Nov 2012 01:01:33 GMT
Server: Manta
x-request-id: 7b872080-3a89-11e2-8d0c-417db534d10b
x-response-time: 8
x-server-name: fb07e9ec-5137-418e-aff2-01d00aff1a49
Connection: keep-alive

SnapLinks

Creates a SnapLink to an object. On success, an HTTP 204 is returned. Specify the "source" object by sending the path in the Location header.

Sample Request

First make an object, then create a link:

manta /$MANTA_USER/stor/foo.json -X PUT -H 'content-type: application/json' \
        -d '{"hello": "world"}'

$ manta /$MANTA_USER/stor/foo.json.2 -X PUT \
     -H 'content-type: application/json; type=link' \
     -H 'Location: /$MANTA_USER/stor/foo.json'

PUT /$MANTA_USER/stor/foo.json.2 HTTP/1.1
Host: us-central.manta.mnx.io
Accept: */*
content-type: application/json; type=link
Location: /$MANTA_USER/stor/foo.json
Date: Fri, 30 Nov 2012 01:14:17 GMT
Authorization: $Authorization


HTTP/1.1 204 No Content
Etag: 447a76c7-c1ff-4613-b64a-02c059a02d92
Last-Modified: Fri, 30 Nov 2012 01:13:59 GMT
Date: Fri, 30 Nov 2012 01:14:18 GMT
Server: Manta
x-request-id: 4391f400-3a8b-11e2-928b-1324a0f99d70
x-response-time: 17
x-server-name: ef69fb79-d88b-4c9d-be81-de55dd60da5b
Connection: keep-alive

Jobs

CreateJob (POST /:login/jobs)

Submits a new job to be executed. This call is not idempotent, so calling it twice will create two jobs. On success, an HTTP 201 is returned with the URI for the job in the Location header.

Inputs

The body of this request must be a JSON document with the following properties:

NameJS TypeDescription
nameString_(optional)_ An arbitrary name for this job
phasesArray(required) tasks to execute as part of this job

phases must be an Array of Object, where objects have the following properties:

NameJS TypeDescription
typeString_(optional)_ one of: map or reduce
assetsArray(String)_(optional)_ an array of objects to be placed in your compute zones
execString(required) the actual (shell) statement to execute
initString(required) shell statement to execute in each compute zone before any tasks are executed
countNumber_(optional)_ an optional number of reducers for this phase (reduce-only): default is 1
memoryNumber_(optional)_ an optional amount of DRAM to give to your compute zone (MB)
diskNumber_(optional)_ an optional amount of disk space to give to your compute zone (GB)

Sample Request

cat job.json
{
    "name": "word count",
    "phases": [ {
        "exec": "wc"
    }, {
        "type": "reduce",
        "exec": "awk '{ l += $1; w += $2; c += $3 } END { print l, w, c }'"
    } ]
}
$ manta /$MANTA_USER/jobs -X POST -H 'content-type: application/json' --data-binary @job.json

POST /$MANTA_USER/jobs HTTP/1.1
Host: us-central.manta.mnx.io
Accept: */*
content-type: application/json
Date: Fri, 30 Nov 2012 01:25:28 GMT
Authorization: $Authorization
Content-Length: 187


HTTP/1.1 201 Created
Content-Length: 0
Location: /$MANTA_USER/jobs/a62ba79e-4d5b-4773-bff9-ecae0fe30dfa
Date: Fri, 30 Nov 2012 01:25:28 GMT
Server: Manta
x-request-id: d3534b60-3a8c-11e2-8d0c-417db534d10b
x-response-time: 10
x-server-name: fb07e9ec-5137-418e-aff2-01d00aff1a49
Connection: keep-alive

AddJobInputs (POST /:login/jobs/:id/live/in)

Submits inputs to an already created job, as created by CreateJob.

Inputs are object names, and are fed in as a \n separated stream (content-type as text/plain). Inputs will be processed as they are received.

An HTTP 204 is returned on success.

Sample Request

note: all lines in inputs.txt have a trailing \n

cat inputs.txt
/$MANTA_USER/stor/words
/$MANTA_USER/stor/words.2
/$MANTA_USER/stor/words.3
$ manta /$MANTA_USER/jobs/a62ba79e-4d5b-4773-bff9-ecae0fe30dfa/live/in -X POST \
        -H content-type:text/plain --data-binary @inputs.txt

POST /$MANTA_USER/jobs/a62ba79e-4d5b-4773-bff9-ecae0fe30dfa/live/in HTTP/1.1
Host: us-central.manta.mnx.io
Accept: */*
content-type: text/plain
Date: Tue, 04 Dec 2012 16:20:17 GMT
Authorization: $Authorization
Content-Length: 55


HTTP/1.1 204 No Content
Connection: close
Date: Tue, 04 Dec 2012 16:20:17 GMT
Server: Manta
x-request-id: 7dc1d950-3e2e-11e2-9dd2-2bc6e71ae3bc
x-response-time: 72
x-server-name: 27721605-e942-4dfb-b01d-9b9ac9847dde

EndJobInput (POST /:login/jobs/:id/live/in/end)

This "closes" input for a job, and will finalize the job. If there are reducers, you likely won't see output until after this is called. There is no input to this API. On success an HTTP 202 is returned.

Sample Request

manta /$MANTA_USER/jobs/a62ba79e-4d5b-4773-bff9-ecae0fe30dfa/live/in/end -X POST

POST /$MANTA_USER/jobs/a62ba79e-4d5b-4773-bff9-ecae0fe30dfa/live/in/end HTTP/1.1
Host: us-central.manta.mnx.io
Accept: */*
Date: Tue, 04 Dec 2012 16:26:24 GMT
Authorization: $Authorization


HTTP/1.1 202 Accepted
Connection: close
Date: Tue, 04 Dec 2012 16:26:24 GMT
Server: Manta
x-request-id: 58b3f570-3e2f-11e2-be8f-87b1bf6e7c11
x-response-time: 27
x-server-name: 8469d848-aee5-4562-957a-38b326ad454f

CancelJob (POST /:login/jobs/:id/live/cancel)

This cancels a job from doing any further work. Cancellation is asynchronous and "best effort"; there is no guarantee the job will actually stop. For example, short jobs where input is already closed will likely still run to completion. This is however useful when:

On success an HTTP 202 is returned.

Sample Request

manta /$MANTA_USER/jobs/495a3099-0394-4904-a17c-5317b5b2162c/live/cancel -X POST

POST /$MANTA_USER/jobs/495a3099-0394-4904-a17c-5317b5b2162c/live/cancel HTTP/1.1
Host: us-central.manta.mnx.io
Accept: */*
content-type: application/json
Date: Tue, 04 Dec 2012 16:34:17 GMT
Authorization: $Authorization

HTTP/1.1 202 Accepted
Connection: close
Date: Tue, 04 Dec 2012 16:34:18 GMT
Server: Manta
x-request-id: 72e68b00-3e30-11e2-be8f-87b1bf6e7c11
x-response-time: 15
x-server-name: 8469d848-aee5-4562-957a-38b326ad454f

ListJobs (GET /:login/jobs)

Returns the list of jobs you currently have. This is a streaming JSON payload that will be identical to listing a directory. Note you can filter the set of jobs down to only live jobs by using the query parameter ?state=running.

On success, an HTTP 200 is returned with a stream of job objects (see CreateJob). Note a content-length header will not be sent back.

Sample Request

Additional newlines added for clarity:

manta /$MANTA_USER/jobs | json -ga

GET /$MANTA_USER/jobs HTTP/1.1
Host: us-central.manta.mnx.io
Accept: */*
Date: Tue, 04 Dec 2012 16:35:56 GMT
Authorization: $Authorization


HTTP/1.1 200 OK
Connection: close
Date: Tue, 04 Dec 2012 16:35:56 GMT
Server: Manta
x-request-id: ada10720-3e30-11e2-9dd2-2bc6e71ae3bc
x-response-time: 5
x-server-name: 27721605-e942-4dfb-b01d-9b9ac9847dde
Content-Type: application/x-json-stream; type=directory
Transfer-Encoding: chunked

{"name":"a1782c7d-529e-4a1a-980d-dfdc805c3124","type":"directory","mtime":"2012-11-29T23:48:00Z"}
{"name":"980c514f-bc59-4fb4-b357-d03f9067dffa","type":"directory","mtime":"2012-11-29T23:48:00Z"}

GetJob (GET /:login/jobs/:id/live/status)

Gets the high-level job container object for a given id. Content-type will be application/json.

An HTTP 204 is returned on success.

Sample Request

Additional newlines added for clarity:

manta /$MANTA_USER/jobs/a62ba79e-4d5b-4773-bff9-ecae0fe30dfa/live/status | json

GET /$MANTA_USER/jobs/a62ba79e-4d5b-4773-bff9-ecae0fe30dfa/live/status HTTP/1.1
Host: us-central.manta.mnx.io
Accept: */*
Date: Tue, 04 Dec 2012 16:41:29 GMT
Authorization: $Authorization


HTTP/1.1 200 OK
Connection: close
Content-Type: application/json
Content-Length: 325
Content-MD5: 6txxiXII5zL+8Ombjc3CwA==
Date: Tue, 04 Dec 2012 16:41:30 GMT
Server: Manta
x-request-id: 7453ba20-3e31-11e2-acef-af63e7e14029
x-response-time: 5
x-server-name: c5df6b59-f27d-4f02-b6db-0e6b044b0d79

{
  "id": "a62ba79e-4d5b-4773-bff9-ecae0fe30dfa",
  "name": "word count",
  "state": "done",
  "cancelled": false,
  "inputDone": true,
  "timeCreated": "2012-12-04T16:23:13.641Z",
  "timeDone": "2012-12-04T16:26:24.998Z",
  "phases": [
    {
      "exec": "wc",
      "type": "map"
    },
    {
      "type": "reduce",
      "exec": "awk '{ l += $1; w += $2; c += $3 } END { print l, w, c }'"
    }
  ]
}

GetJobOutput (GET /:login/jobs/:id/live/out)

Returns the current "live" set of outputs from a job. Think of this like tail -f. The objects are returned as a stream, with an HTTP status code of 200 on success.

Sample Request

manta /$MANTA_USER/jobs/a62ba79e-4d5b-4773-bff9-ecae0fe30dfa/live/out

GET /$MANTA_USER/jobs/a62ba79e-4d5b-4773-bff9-ecae0fe30dfa/out HTTP/1.1
Host: us-central.manta.mnx.io
Accept: */*
Date: Tue, 04 Dec 2012 16:47:14 GMT


HTTP/1.1 200 OK
Connection: close
Content-Type: text/plain
Result-Set-Size: 1
Date: Tue, 04 Dec 2012 16:47:17 GMT
Server: Manta
x-request-id: 43745300-3e32-11e2-9dd2-2bc6e71ae3bc
x-response-time: 18
x-server-name: 27721605-e942-4dfb-b01d-9b9ac9847dde
Transfer-Encoding: chunked

/$MANTA_USER/jobs/a62ba79e-4d5b-4773-bff9-ecae0fe30dfa/stor/reduce.1.652bdcac-28c8-4e07-b887-91f513966e8b

At this point you could retrieve that object (only data shown):

manta /$MANTA_USER/jobs/a62ba79e-4d5b-4773-bff9-ecae0fe30dfa/stor/reduce.1.652bdcac-28c8-4e07-b887-91f513966e8b
75432 75432 620022

GetJobInput (GET /:login/jobs/:id/live/in)

Submitted input objects are available while a job is running as a \n separated text stream (the output will simply be the object's name). The objects are returned as a stream, with an HTTP status code of 200 on success.

Sample Request

manta /$MANTA_USER/jobs/a62ba79e-4d5b-4773-bff9-ecae0fe30dfa/live/in

GET /$MANTA_USER/jobs/a62ba79e-4d5b-4773-bff9-ecae0fe30dfa/live/in HTTP/1.1
Host: us-central.manta.mnx.io
Accept: */*
Date: Tue, 04 Dec 2012 16:52:05 GMT
Authorization: $Authorization


HTTP/1.1 200 OK
Connection: close
Content-Type: text/plain
Result-Set-Size: 3
Date: Tue, 04 Dec 2012 16:52:05 GMT
Server: Manta
x-request-id: ef3d8710-3e32-11e2-9dd2-2bc6e71ae3bc
x-response-time: 13
x-server-name: 27721605-e942-4dfb-b01d-9b9ac9847dde
Transfer-Encoding: chunked

/$MANTA_USER/stor/words
/$MANTA_USER/stor/words.2
/$MANTA_USER/stor/words.3

GetJobFailures (GET /:login/jobs/:id/live/fail)

Returns the current "live" set of failures from a job. Think of this like tail -f. The objects are returned as a stream, with an HTTP status code of 200 on success.

Sample Request

manta /$MANTA_USER/jobs/d558881d-a02f-43d3-846b-84771b6bf1ed/live/fail

GET /$MANTA_USER/jobs/d558881d-a02f-43d3-846b-84771b6bf1ed/live/fail HTTP/1.1
Host: us-central.manta.mnx.io
Accept: */*
Date: Tue, 04 Dec 2012 16:56:14 GMT
Authorization: $Authorization


HTTP/1.1 200 OK
Connection: close
Content-Type: text/plain
Date: Tue, 04 Dec 2012 16:56:15 GMT
Server: Manta
x-request-id: 83e98670-3e33-11e2-be8f-87b1bf6e7c11
x-response-time: 6
x-server-name: 8469d848-aee5-4562-957a-38b326ad454f
Transfer-Encoding: chunked

/$DIFFERENT_MANTA_USER/stor/words
/$MANTA_USER/stor/words.4

GetJobErrors (GET /:login/jobs/:id/live/err)

Returns the current "live" set of errors from a job. Think of this like tail -f. The objects are returned as a stream, with an HTTP status code of 200 on success. The body is a JSON stream application/x-json-stream; type=job-error.

NameTypeDescription
taskIdStringunique identifier for the task
phaseNumNumberphase number of the failure
whatStringa human readable summary of what failed
codeStringprogrammatic error code
messageStringhuman readable error message
serverStringlogical server name where the task was executed
machineStringzonename where the task was executed
inputString(optional) object being processed when the map task failed
p0inputString(optional) phase 0 input object, returned if not preceded by reduce phase
coreString(optional) core file name, if any process in the task dumped core
stderrString(optional) a key that saved the stderr for the given command

Sample Request

Additional newlines added for clarity:

manta /$MANTA_USER/jobs/d558881d-a02f-43d3-846b-84771b6bf1ed/live/err | json -ga

GET /$MANTA_USER/jobs/d558881d-a02f-43d3-846b-84771b6bf1ed/live/err HTTP/1.1
Host: us-central.manta.mnx.io
Accept: */*
Date: Tue, 04 Dec 2012 17:14:43 GMT
Authorization: $Authorization

HTTP/1.1 200 OK
Connection: close
Content-Type: application/x-json-stream; type=job-error
Date: Tue, 04 Dec 2012 17:14:44 GMT
Server: Manta
x-request-id: 190329d0-3e36-11e2-acef-af63e7e14029
x-response-time: 13
x-server-name: c5df6b59-f27d-4f02-b6db-0e6b044b0d79
Transfer-Encoding: chunked

{
  "id": "d558881d-a02f-43d3-846b-84771b6bf1ed",
  "phase": 0,
  "what": "phase 0: map key \"/$DIFFERENT_MANTA_USER/stor/words\"",
  "code": "EJ_ACCES",
  "message": "permission denied: \"/$DIFFERENT_MANTA_USER/stor/words\"",
  "key": "/$DIFFERENT_MANTA_USER/stor/words"
}
{
  "id": "d558881d-a02f-43d3-846b-84771b6bf1ed",
  "phase": 0,
  "what": "phase 0: map key \"/$MANTA_USER/stor/words.4\"",
  "code": "EJ_NOENT",
  "message": "no such object: \"/$MANTA_USER/stor/words.4\"",
  "key": "/$MANTA_USER/stor/words.4"
}

Multipart Uploads

CreateMultipartUpload (POST /:login/uploads)

Creates a new mulitpart upload. This call is not idempotent, so calling it twice, even with the same inputs, will create two mulitpart uploads. On success, an HTTP 201 is returned with a JSON document containing the URI for the multipart upload and its ID in the response body.

Inputs

The body of this request must be a JSON document with the following properties:

NameJS TypeDescription
objectPathString(required) the Manta path where the target object will be stored
headersArray(String)(optional) an array of headers to store on the target object

Headers supplied in the request body will be stored as headers on the target object when the multipart upload is committed. Headers may only be provided when the multipart upload is created and cannot be changed until after the target object has been committed. As with all Manta objects, you are allowed 4 KB of header data.

The following headers are supported for target objects:

The following headers are not supported and will cause the request to fail with a MultipartUploadInvalidArgumentError:

Unsupported headers may change in future revisions.

Sample Request

To use the sample request data below exactly, replace $MANTA_USER with your Manta login.

cat mpu.json
{
    "objectPath": "/$MANTA_USER/stor/myObj.txt",
    "headers": {
        "content-length": 5242900,
        "content-md5": "e1iePOxoKQYqJS6MLu6lyg=="
    }
}

$ manta /$MANTA_USER/uploads -X POST -H 'content-type: application/json' --data-binary @mpu.json

POST /$MANTA_USER/uploads HTTP/1.1
Host: us-central.manta.mnx.io
User-Agent: curl/7.51.0
Accept: */*
content-type: application/json
date: Tue, 09 Jan 2018 22:44:29 GMT
Authorization: $Authorization
Content-Length: 161


HTTP/1.1 201 Created
Connection: close
Content-Type: application/json
Content-Length: 125
Content-MD5: xrJUHTD0ZjzZUZElJdezDA==
Date: Tue, 09 Jan 2018 22:44:30 GMT
Server: Manta
x-request-id: a73fb1e0-f58e-11e7-83c3-f3f97664d32b
x-response-time: 453
x-server-name: 9857d980-a49b-c317-b313-d1649b6ed333

{
    "id":"d52da3a7-a55c-4961-b95a-84fbfc5d7903",
    "partsDirectory":"/$MANTA_USER/uploads/d52/d52da3a7-a55c-4961-b95a-84fbfc5d7903"
}

UploadPart (PUT /:login/uploads/[0-f]+/:id/:partNum)

Creates or overwrites a part for a multipart upload. This operation generally functions like the PutObject operation, with some key differences highlighted below. On success, an HTTP 204 is returned.

As with PutObject, an etag for the part is returned in the "Etag" header of the response. The service uses etags and part numbers to identify parts when the multipart upload is committed, so you should save this value to use for the CommitMultipartUpload operation.

The service will store the number of copies specified in the Durability-Level header input to CreateMultipartUpload when the multipart upload was created, or two copies if no durability level was specified. As with all target object headers, the durability level cannot be changed once a multipart upload is created.

As with the PutObject operation, you may specify a Content-MD5 header for the part, which the service will verify matches the data uploaded. Unlike PutObject, conditional request semantics (e.g., If-Match or If-Modified-Since) are not supported for UploadPart.

The multipart upload API requires a minimum part size of 5 MB for all parts except the last one. This size is validated when CommitMultipartUpload is called. UploadPart operations will succeed even if the part is under the minimum size.

Any part number between 0 and 9999 is valid, but committing a multipart upload requires consecutive parts starting from part 0. See CommitMultipartUpload for details.

Sample Request

manta /$MANTA_USER/uploads/d52/d52da3a7-a55c-4961-b95a-84fbfc5d7903/0 \
    -X PUT -H 'content-type: text/plain' --data-binary @5mb-part.txt

PUT /$MANTA_USER/uploads/d52/d52da3a7-a55c-4961-b95a-84fbfc5d7903/0
HTTP/1.1
Host: us-central.manta.mnx.io
User-Agent: curl/7.51.0
Accept: */*
content-type: text/plain
date: Wed, 10 Jan 2018 22:02:51 GMT
Authorization: $Authorization
Content-Length: 5242880
Expect: 100-continue


HTTP/1.1 100 Continue


HTTP/1.1 204 No Content
Connection: close
Etag: 92a9e0de-9498-46bf-b559-0da424a5ea77
Last-Modified: Wed, 10 Jan 2018 22:02:52 GMT
Computed-MD5: XzY+DlipXwbL6bvGYsXftg==
Date: Wed, 10 Jan 2018 22:02:52 GMT
Server: Manta
x-request-id: 00a2f440-f652-11e7-958f-0db50252a777
x-response-time: 633
x-server-name: 9857d980-a49b-c317-b313-d1649b6ed333

GetMultipartUpload (GET /:login/uploads/[0-f]+/:id/state)

Gets the high-level multipart upload container object for a given id. Content-type will be application/json. An HTTP 204 is returned on success.

Sample Request

manta /$MANTA_USER/uploads/d52/d52da3a7-a55c-4961-b95a-84fbfc5d7903/state | json

GET /$MANTA_USER/uploads/d52/d52da3a7-a55c-4961-b95a-84fbfc5d7903/state
HTTP/1.1
Host: us-central.manta.mnx.io
User-Agent: curl/7.51.0
Accept: */*
date: Wed, 10 Jan 2018 22:08:51 GMT
Authorization: $Authorization


HTTP/1.1 200 OK
Connection: close
Content-Type: application/json
Content-Length: 311
Content-MD5: Gy/GKwCzvvOt2NuQXxfOkg==
Date: Wed, 10 Jan 2018 22:08:52 GMT
Server: Manta
x-request-id: d762ff70-f652-11e7-958f-0db50252a777
x-response-time: 267
x-server-name: 9857d980-a49b-c317-b313-d1649b6ed333


{
     "id": "d52da3a7-a55c-4961-b95a-84fbfc5d7903",
     "state": "created",
     "partsDirectory": "/$MANTA_USER/uploads/d52/d52da3a7-a55c-4961-b95a-84fbfc5d7903",
     "targetObject": "/$MANTA_USER/stor/myObj.txt",
     "headers": {
         "content-length": 5242900,
         "content-md5": "e1iePOxoKQYqJS6MLu6lyg=="
     },
     "numCopies": 2,
     "creationTimeMs": 1515537870145
}

CommitMultipartUpload (POST /:login/uploads/[0-f]+/:id/commit)

Commits the multipart upload with the input set of parts. This operation is atomic and idempotent. Once a multipart upload is committed, it cannot be aborted or committed with a different set of parts, though the operation may be safely retried with the same inputs. On success, an HTTP 201 is returned with the URI of the target object in the Location header. The URI of the target object is the same as the "objectPath" specified when the multipart upload was created.

Inputs

The body of this request must be a JSON document with the following properties:

NameJS TypeDescription
partsArray(String)(required) array of etags of parts to commit, in order, starting from part 0

The service will verify that the part etags specified in the input array match what is currently in Manta and that all parts, except the last part, meet the minimum part size of 5 MB. Additionally, the service will verify that the caller has permissions to operate on the "objectPath" URI specified when the multipart upload was created.

Sample Request

cat commit.json
{
    "parts": [
        "92a9e0de-9498-46bf-b559-0da424a5ea77",
        "e1bbee18-72e8-409d-ad75-b359989e8c1a"
    ]
}

$ manta /$MANTA_USER/uploads/af1/af143a31-c332-4442-a9a9-773894fe98c3/commit -X POST \
    -H 'content-type: application/json' --data-binary @commit.json

POST /$MANTA_USER/uploads/af1/af143a31-c332-4442-a9a9-773894fe98c3/commit
HTTP/1.1
Host: us-central.manta.mnx.io
User-Agent: curl/7.51.0
Accept: */*
content-type: application/json
date: Wed, 10 Jan 2018 22:33:28 GMT
Authorization: $Authorization
Content-Length: 120


HTTP/1.1 201 Created
Connection: close
Etag: 68b5778d-85a9-4529-9bfd-fb9fe52e3b3a
Last-Modified: Wed, 10 Jan 2018 22:33:25 GMT
Location: /$MANTA_USER/stor/myObj.txt
Computed-MD5: e1iePOxoKQYqJS6MLu6lyg==
Date: Wed, 10 Jan 2018 22:33:29 GMT
Server: Manta
x-request-id: 4771cc30-f656-11e7-958f-0db50252a777
x-response-time: 994
x-server-name: 9857d980-a49b-c317-b313-d1649b6ed333
Transfer-Encoding: chunked

AbortMultipartUpload (POST /:login/uploads/[0-f]+/:id/abort)

Aborts the multipart upload. This operation is atomic and idempotent. Once a multipart upload has been aborted, it cannot be committed, but the AbortMultipartUpload operation may safely be retried. On success, an HTTP 204 is returned.

Sample Request

manta /$MANTA_USER/uploads/d52/d52da3a7-a55c-4961-b95a-84fbfc5d7903/abort -X POST

POST /$MANTA_USER/uploads/d52/d52da3a7-a55c-4961-b95a-84fbfc5d7903/abort
HTTP/1.1
Host: us-central.manta.mnx.io
User-Agent: curl/7.51.0
Accept: */*
date: Wed, 10 Jan 2018 22:14:01 GMT
Authorization: $Authorization


HTTP/1.1 204 No Content
Connection: close
Date: Wed, 10 Jan 2018 22:14:02 GMT
Server: Manta
x-request-id: 90376f40-f653-11e7-958f-0db50252a777
x-response-time: 308
x-server-name: 9857d980-a49b-c317-b313-d1649b6ed333

Multipart Upload Redirect Endpoints

To allow clients to resolve a multipart upload's URI based only on its upload ID, requests to /:login/uploads/:id and /:login/uploads/:id/:partNum will redirect the request to the multipart upload's fully qualified parts directory. On success, an HTTP 301 is returned, with the parts directory URI in the "Location" response header.

Sample Request

manta /$MANTA_USER/uploads/d52da3a7-a55c-4961-b95a-84fbfc5d7903

GET /$MANTA_USER/uploads/d52da3a7-a55c-4961-b95a-84fbfc5d7903 HTTP/1.1
Host: us-central.manta.mnx.io
User-Agent: curl/7.51.0
Accept: */*
date: Wed, 10 Jan 2018 23:15:49 GMT
Authorization: $Authorization


HTTP/1.1 301 Moved Permanently
Connection: close
Location: /$MANTA_USER/uploads/d52/d52da3a7-a55c-4961-b95a-84fbfc5d7903
Date: Wed, 10 Jan 2018 23:15:50 GMT
Server: Manta
x-request-id: 325fc210-f65c-11e7-958f-0db50252a777
x-response-time: 141
x-server-name: 9857d980-a49b-c317-b313-d1649b6ed333
Transfer-Encoding: chunked

If the multipart upload does not exist, the redirect endpoints will return an HTTP 404:

manta /$MANTA_USER/uploads/1afb5e32-ae0e-4b4a-b4db-efaa3c694527 | json

GET /$MANTA_USER/uploads/1afb5e32-ae0e-4b4a-b4db-efaa3c694527 HTTP/1.1
Host: 172.27.13.143:8080
Host: us-central.manta.mnx.io
User-Agent: curl/7.51.0
Accept: */*
date: Wed, 10 Jan 2018 23:19:25 GMT
Authorization: $Authorization


HTTP/1.1 404 Not Found
Connection: close
Content-Type: application/json
Content-Length: 110
Content-MD5: OwMLZJnfs9GTFuLlfsVOAg==
Date: Wed, 10 Jan 2018 23:19:26 GMT
Server: Manta
x-request-id: b30da710-f65c-11e7-958f-0db50252a777
x-response-time: 66
x-server-name: 9857d980-a49b-c317-b313-d1649b6ed333

{
    "code": "ResourceNotFound",
    "message": "/$MANTA_USER/uploads/1afb5e32-ae0e-4b4a-b4db-efaa3c694527 was not found"
}

The complete list of redirect endpoints is as follows: