Author Archives: Raghavendar T S

cannot change DocValues type from SORTED to SORTED_SET – DataStax Graph – Titan – Solr

cannot change DocValues type from SORTED to SORTED_SET – DataStax Graph – Titan – Solr

Are you facing these errors???

  1. cannot change docvalues type from sorted_set to sorted for field field_name
  2. cannot change docvalues type from sorted to sorted_set for field field_name

No worries. We have the solution. First of all we need to know what DocValues means in Solr. You will find a lot of information in the internet. Now we will see why we are getting the error “cannot change DocValues type from SORTED_SET to SORTED for field field_name”.

Lets us consider we have a field named description with type solr.TextField with docValues property true. We will now have a separate data stored which is meant for performing efficient faceting, sorting and other operations. In order to support search requirements, we want to change the type to some other existing Solr supported types or even custom type. So basically we will update the vertex label schema and reindex the data. In this process, Solr will try to reindex the data with new schema changes. What If the field data values of the older field type is not compatible with the newer field type? Yes we get error in this case. So what is the solution? 

Solution:

  1. Update the field data type and disable DocValues
  2. Reindex Data
  3. Enable DocValue
  4. Reindex Data
  5. That’s it

Steps:

Step 1:

Execute the following queries in the server.

dsetool get_core_config {{GRAPH_NAME}}.{{VERTEX_LABEL}}_p > {{VERTEX_LABEL}}_config.xml

dsetool get_core_schema {{GRAPH_NAME}}.{{VERTEX_LABEL}}_p > {{VERTEX_LABEL}}_schema.xml

E.g.

dsetool get_core_config TESTGRAPH.TESTLABEL_p > TESTLABEL_config.xml

dsetool get_core_schema TESTGRAPH.TESTLABEL_p > TESTLABEL_schema.xml

The above two queries will create a config file and a schema file for the given vertex label. Take backup of the config and schema file.

Step 2:

Open the schema file TESTLABEL_schema.xml. You will find some similar to the following line.

<field docValues="true" indexed="true" multiValued="false"
name="description" stored="true" type="StrField"/>

Update the docValues property to false and save the file. Make sure you update the field data type which you want to change which is description in our case.

Step 3:

Execute the following command which will clear the existing docValues data.

dsetool reload_core TESTGRAPH.TESTLABEL_p reindex=true
schema=TESTLABEL_schema.xml solrconfig=TESTLABEL_config.xml

Step 4:

Open the schema file TESTLABEL_schema.xml. Update the docValues property to true and save the file.

Step 5:

Execute the following command which will create the required docValues data.

dsetool reload_core TESTGRAPH.TESTLABEL_p reindex=true
schema=TESTLABEL_schema.xml solrconfig=TESTLABEL_config.xml

Step 6:

Enjoy!!!

Chrome Elasticsearch Sense Not Working – Solved

Chrome Elasticsearch Sense Not Working – Solved

Sense plugin has been marked as malware by Google and so the same has been removed from the Chrome Web Store. We have a work around to install Kibana which has Sense builtin. But If you do not want to break your head, we have an option to install Sense plugin by enabling Developer mode in Extensions.

Steps:

  1. Download and extract the zip file.
  2. Click the Chrome menu => More tools => Extensions
  3. Enable the checkbox Developer mode
  4. Click Load unpacked extension
  5. Select the extracted folder.
  6. Enjoy

Writing REST APIs – CREATE/POST – Best Practices – Part II

Writing REST APIs – CREATE/POST – Best Practices – Part II

This is my second post on Writing REST APIs. Today the focus is on POST HTTP method. Basically POST method is used for create a resource.

Sample Request :

POST /users
{
     "users" : [{
          "firstName" : "Tom",
          "lastName" : "Cruise"
     },{
          "firstName" : "Tom",
          "lastName" : "Hardy"
     }]
}

Sample Response :

{
     "statusCode" : 201,
     "statusDescr" : "Created"
     "details" : [{
          "id" : "7995af4b-405e-48a2-960e-3d2f2918db0e",
          "statusCode" : 201,
          "statusDescr" : "Created"
     },{
          "id" : "e9768b26-097c-4b65-b1f5-dddcdc2790b4",
          "statusCode" : 201,
          "statusDescr" : "Created"
     }]
}

The id field in the response is UUID which will uniquely identify the created resource. We will be using this UUID to access this resource using PUT or GET. In each of the PUT or GET request, what we can do is that we can validate the id field to check whether the id is a valid UUID string.

  UUID.fromString("SAMPLE UUID");

The above snippet will throw Exception If we fail to pass a valid UUID. So what is the use? Let’s say we have created a resource in database which has id field with some valid UUID string. We will do a GET request with this id which is a valid case. Now we do a GET request with invalid id (some random id string). It is for sure that the resource will not be in the database since the id which we are using is not a valid UUID. So in this case we are doing an unnecessary DB call which we can reduce by validating id field. REST APIs are developed for it to be consumed by anyone including people (using any REST Client), web applications, real-time processors (may be Storm or Spark) etc. In few cases, there are possibilities for the REST APIs to be called with invalid id field and so it is better to validate the id field. In an application, we will have usecases where we will be using UUID in multiple places like accountId, userId, etc and we can validate these field from the request to avoid unnecessary database calls in invalid requests.

Default Fields/Attributes:

When we are creating a resource, it is better to have the following field stored in the database.

  • Created Date – Resource created date
  • Updated Date – Recently updated date of the resource
  • User ID – User ID of the person who recently updated the resource
  • Version – Version of the API with which we created the resource
  • Application ID

We will see a little more information on version field. Let’s say we have released User API for the first time and URL for it is /API/users/v1. After a few weeks/months we release a enhanced version of the API and the new URL is /API/users/v2. Note that we cannot update the existing version of the API unless it doesn’t break the consumers applications. In this case, the version field will store v1 or v2.

What is Application ID? REST APIs will be consumed by any application and these application will call the REST APIs using a  authentication token (used in HTTP header) with which we will be able to identify who is calling the REST API and this is how most of the REST APIs will be designed.  So Application ID will store the ID of the application which is the consuming the REST API.

Related Links:

  1. Writing REST APIs - CRUD - Request & Response - Best Practices - Part I
  2. Writing REST APIs - CREATE/POST - Best Practices – Part II
Read More...

Writing REST APIs – CRUD – Request & Response – Best Practices – Part I

Writing REST APIs - CRUD - Request & Response - Part I

Writing REST APIs – CRUD – Request & Response – Best Practices – Part I

I am going to publish a series of posts  related to REST APIs standards and today we will see some details on designing the request and response for an API. So now we will assume we are going to write an API for user managements and we call this User API. When writing an API, the following are the 5 basic endpoint required to supports most of the common use-cases.

  1. POST – Create one or more users
  2. PATCH – Update one user by given ID
  3. PATCH – Update one or more users by given ID’s
  4. GET – Get user by given ID
  5. GET – Get all users
  6. DELETE – Delete user by given ID

No we will see in detail about the request and response structure.


1 . POST – Create one or more users

Request :

POST /users
{
     "users" : [{
          "firstName" : "Tom",
          "lastName" : "Cruise"
     },{
          "firstName" : "Tom",
          "lastName" : "Hardy"
     }]
}

Response 1 :

{
     "statusCode" : 201,
     "statusDescr" : "Created"
     "details" : [{
          "id" : "xxxxxxxxxxxxxxxxxxxxxxxxx",
          "statusCode" : 201,
          "statusDescr" : "Created"
     },{
          "id" : "yyyyyyyyyyyyyyyyyyyyyyyyy",
          "statusCode" : 201,
          "statusDescr" : "Created"
     }]
}

Response 2 :

If you notice in the below response, we have different responses and so we have the global response to be 207 (Multi-Status) HTTP status code.

{
     "statusCode" : 207,
     "statusDescr" : "Multi-Status"
     "details" : [{
          "id" : "xxxxxxxxxxxxxxxxxxxxxxxxx",
          "statusCode" : 201,
          "statusDescr" : "Created"
     },{
          "id" : "yyyyyyyyyyyyyyyyyyyyyyyyy",
          "statusCode" : 201,
          "statusDescr" : "Conflict"
     }]
}

2. PATCH – Update one user by given ID

Request :

PATCH /users/xxxxxxxxxxxxxxxxxxxxxxxxx
{
     "firstName":"Tom"
}

Response 1 : 

HTTP Response 200
No response body for successful update

Response 2 : Error Case

{
     "statusCode" : 400,
     "statusDescr" : "PATCH not supported on the field firstName"
}

3. PATCH – Update one or more users by given ID’s

Request :

PATCH /users
{
     "users" : [{
          "id" : "xxxxxxxxxxxxxxxxxxxxxxxxx"
          "firstName" : "John",
     },{
          "id" : "yyyyyyyyyyyyyyyyyyyyyyyyy"
          "firstName" : "Sam",
     }]
}

Response 1 :

{
     "statusCode" : 200,
     "statusDescr" : "OK"
     "details" : [{
          "id" : "xxxxxxxxxxxxxxxxxxxxxxxxx",
          "statusCode" : 200,
          "statusDescr" : "OK"
     },{
          "id" : "yyyyyyyyyyyyyyyyyyyyyyyyy",
          "statusCode" : 200,
          "statusDescr" : "OK"
     }]
}

Response 2 :

If you notice in the below response, we have different responses and so we have the global response to be 207 (Multi-Status) HTTP status code.

{
     "statusCode" : 207,
     "statusDescr" : "Multi-Status"
     "details" : [{
          "id" : "xxxxxxxxxxxxxxxxxxxxxxxxx",
          "statusCode" : 200,
          "statusDescr" : "OK"
     },{
          "id" : "yyyyyyyyyyyyyyyyyyyyyyyyy",
          "statusCode" : 400,
          "statusDescr" : "PATCH not supported on the field firstName"
     }]
}

4. GET – Access user by given ID

Request :

GET /users/xxxxxxxxxxxxxxxxxxxxxxxxx

Response 1 :

{
     "id" : "xxxxxxxxxxxxxxxxxxxxxxxxx"
     "firstName" : "Tom",
     "lastName" : "Cruise"
}

Response 2 :  Error Case

{
     "statusCode" : 404,
     "statusDescr" : "Not Found"
}

5. GET – Get all users

Request :

GET /users

Response 1 :

{
     "users" :[{     
          "id" : "xxxxxxxxxxxxxxxxxxxxxxxxx"
          "firstName" : "Tom",
          "lastName" : "Clancy"
     },{     
          "id" : "yyyyyyyyyyyyyyyyyyyyyyyyy"
          "firstName" : "Tom",
          "lastName" : "Cruise"
     }]
}

Response 2 :

{
     "users" :[]
}

6. DELETE – Delete user by given ID

Request :

DELETE /users/xxxxxxxxxxxxxxxxxxxxxxxxx

Response 1 :

HTTP Response 200
No response body for successful update

Response 2 : Error Case

{
     "statusCode" : 404,
     "statusDescr" : "Not Found"
}

PUT is similar to PATCH operation except that PUT should be used to update (full update) the given resource/user, where as PATCH is used for partial update. The request and response structure remain similar for PATCH and PUT.

Related Links:

  1. Writing REST APIs - CRUD - Request & Response - Best Practices - Part I
  2. Writing REST APIs - CREATE/POST - Best Practices – Part II
Read More...