Implementation of Product Importer in AEM

Reading Time: 6 minutes

Nowadays, every serious company has options on its website to present the products to potential customers. When we talk about big companies, with a lot of products and huge traffic, AEM is one of the best solutions. But, how are the products imported in AEM, where are they placed in AEM are?… you can learn here. We will cover the following aspects of the problem:

  • How to fetch the products from the server
  • How to convert server response (JSON String) into Java object
  • Where to place products in AEM repository
  • Product node structure
  • How to persist in products
  • How to start product importer and follow the process

How to fetch the products from the server

Typically large companies are keeping their products on a separate dedicated server. With the API that they will provide to you, a connection to the server will be established, and you could fetch the products. In most cases, an OSGI service is created that keeps the configuration data for connecting the remote server. Typically we got the response as a JSON String. Bellow is just one idea of how to get a response. The URL parameter and the access token is provided by the client, and usually, we keep it in OSGI service configuration.

private String getAPIResponse(String url) {
		String accessToken = getAccessToken();
		CloseableHttpClient httpclient = HttpClients.createDefault();
		String authorizationString = "Bearer " + accessToken;
		HttpGet request = new HttpGet(URI.create(url));
		request.addHeader("Authorization", authorizationString);
		try {
			HttpResponse response = httpclient.execute(request);
			return getResponseBodyAsString(response);
		} catch (IOException e) {
			LOGGER.error("Failed httpclient.execute method", e);
		}
		return null;
	}
private String getResponseBodyAsString(HttpResponse response) {
		try (Scanner sc = new Scanner(response.getEntity().getContent())) {
			StringBuilder sb = new StringBuilder();
			while (sc.hasNext()) {
				sb.append(sc.nextLine());
			}
			return sb.toString();
		} catch (IOException e) {
			LOGGER.error("Failed to retreive response body", e);
		}
		return null;
	}

How to convert server response into Java object (APIModel)

Once we got a response as JSON String, the biggest challenge is converting response from the server (String apiResponse) into java class (APIModel). For that purpose we use the com.google.gson.Gson class. Sometimes it is unpredictable how Gson will de-serialize apiResponse into java objects. As for advice, if something goes wrong in mapping, just put “Object” in the mapping of the value, and later when debugging can check how Gson actually maps that value.

public APIModel convertIntoAPIModel(String apiResponse) {
 try {
 Gson gson = new Gson();
 return gson.fromJson(apiResponse, APIModel.class);
 } catch (RuntimeException e) {
 LOGGER.error("Error while converting into APIModel",e);
 throw e;
 }
}
@Model(adaptables = Resource.class)
public class APIModel {
	private List<ProductImportedModel> results;

	public List<ProductImportedModel> getResults() {
		return results;
	}
}
public class ProductImportedModel {
     private String nameOfProduct;
     private Date lastModifiedAt;
     private Date createdAt;
     ........
     ........
}

Where to place products in AEM repository

First, let’s look at the very base of this commerce part. We will look at the repository level in CRX to see the location and structure of the products in AEM. For that purpose most appropriate is an out of the box solution for we-retail which is part of the AEM installation. As we can see, products are stored in /var/commerce/products/your-company-name.

Product node structure

Let’s check the structure of one product in we-retail (on the image above “eqbisucos”). The product consists of one “master” product which contains the general properties of the product. These properties can be anything including price, rating, origin… and the most important properties are these two which mark it as a product:

  • cq:commerceType String product
  • sling:resourceType String commerce/components/product

Under this master node, there are subnodes as “image” and variants of the products. Regarding variants, it is important to mention that the difference with the product is that property commerceType has the value ‘variant’.

In the image above , we can see different variants as size-s, size-m, size-l.

Now, when we know the structure of the out of the box commerce product, let’s see how we can use our APIModel and transform it into node structure under /var/commerce.

The structure of the product node is not strictly defined. It depends on the concrete situation and the data for the product that we need to store. However, there are some rules to take in consideration:

  • define the master product node
  • create variants as sub-nodes of the master node. It could happen both (master and variant) to have very similar properties with a small difference, but that is acceptable. At least one property must be different
  • It is not required but would be nice to have “image” sub-nodes with an image for the product. It could be just one “image node” for master, or furthermore, every variant can have its own “image” node
  • It is possible to have other sub-nodes with different information for product or variants. The number of these “other nodes” is not limited. They can keep any information
  • Node structure of master or variant is not required to be the same for every product. Some sub-nodes could be missing for some masters or variants

How to persist products

Once we have determinate the node structure of the product, it is time to create node structure and store values as node properties. As first, for that purpose, we need service user with write permissions in /var/commerce part. The best approach is to use sling API, with all methods to create resources. Here is one example to create product node with properties.

Map<String, Object> properties = new HashMap<>();
		properties.put("sling:resourceType", "commerce/components/product");
		properties.put("cq:commerceType", commerceType);
		properties.put("jcr:primaryType", "nt:unstructured");
		properties.put("price", 1000);
		properties.put("color", "green);
Resource newProduct = resolver.create(productsResource, "myFirstProduct", properties);

How to start product importer and follow the process

Product importer is triggered from AEM JMX console. Possible options are to trigger it manually or periodic as a cron job. It is a separate thread that does not return a result, so it is very hard to follow the process without server logs. But this is possible just for the developer. So, any solution? Yes, best practice would be during the implementation of the importer, when exception happens, that exception to be mailed to the responsible person, with time and the exact point where exception happens.

Tool Showcase: Node-RED

Reading Time: 5 minutes

Node-RED is a flow-oriented tool to wire together hardware devices, APIs and online services. It is mainly targeting the IoT market but can be used for a lot of other things as well. Because of its easy to use browser-based UI and drag and drop programming system, it’s really beginner-friendly with a steep learning curve.

Even though it was developed by IBM in 2013, it’s not really known to most of the IT community. At least none of my colleagues knew it. That’s worth for me to write this tool showcase. Enjoy reading!

Getting started

Instead of just reading along, I encourage everybody to just start Node-RED and try it yourself. If docker is installed, this is just a matter of seconds. Use the following command to start a Node-RED instance locally:

docker run -it -p 1880:1880 --name mynodered nodered/node-red

That’s it. You are ready to go! Open your browser and go to localhost:1880 to access the Node-RED UI.

One of the simplest flows is the following one:

Drag an “http in”, “template” and “http out” node into the flow and connect them. After clicking the deploy button you can access localhost:1880/<whateverPathYouConfiguredInYourHttpInNode> to see whatever you’ve configured in your template node. Congratulations, you have just created your first Node-RED flow!

Of course, rendering static content on an endpoint is not the most exciting thing to do, but between the HTTP in and out nodes, you’re free to do whatever you want. Nodes to make HTTP calls to different URLs, reading and writing files and much more are provided by Node-RED by default.

The power of the community

Node-RED uses Node.js for its nodes (yes, the terminology “node” is overused in the Node-RED context 🙂 ). This has a big advantage, that new nodes can be added easily from the node package manager (npm). You can find these nodes by searching for “node-red-contrib” in the npm repository. An even simpler option is to install these nodes using the “Manage Palette” option in the UI. There you can install new nodes with a single click.

There are nodes for almost everything. Need support for slack? Yep, there’s a node for that. Tired of pressing light switches in your house to turn off and on your Philips Hue lights? Yep, there’s a node for that as well. Whatever you can imagine, there’s a node for it.

A slightly more advanced flow

To test some Node-RED features, I tried to come up with a slightly more complicated example. I own some Philips Hue lamps and a LaMetric Time. After searching some nodes for these devices, I was really surprised that somebody already built some for these two devices (I was especially surprised about the support for the not so well-known LaMetric Time).

My use case was pretty straight forward. Turn on the lights when it gets dark and display a message on my LaMetric near my TV. At midnight, turn off the lights and display a different message. Additionally, I wanted some web endpoints that I could call to trigger both actions manually.

After only a few minutes, I had the following flow:

And it works! I found a node that sends an event as soon as the sun goes down at my particular location. Very cool. All the other nodes (integration for Philips Hue and LaMetric) can also easily be added with the “Manage Palette” option in the GUI. All in all, the implementation of my example use-case was pretty straight forward and required no programming know-how.

Expandability

Even though there are almost 3000 community-contributed nodes available to use, you might have some hardware or API that does not (yet) have some pre-made nodes. In that case, you can implement your own nodes pretty easily. The only thing required is a text editor and some node.js know-how.

The Node-RED documentation provides a good guide on how to create custom nodes: https://nodered.org/docs/creating-nodes/first-node

It is highly recommended to push your custom nodes to the npm repository to be used by the community.

Additional Resources

There are a whole lot more features that are not described in this blogpost.

  • Flows are just .json files and can easily be imported or exported or added to a git repository
  • Flows can be converted to subflows and used like nodes in other flows
  • Multiple flows can run in parallel and trigger each other
  • There are special nodes for error handling or low-level TCP communication
  • There are keyboard shortcuts for everything
  • … and much more!

Feel free to have a look yourself:

Thanks for reading!

Experiences of FrontendConnect 2019 conference Warsaw, Poland

Reading Time: 4 minutes

INTRODUCTION

Everybody has an open lifetime book full of blank pages, waiting to be filled. We write the story as we go, so back in November 2019, I have started the chapter ‘Frontend conferences’ by attending the FrontendConnect2019 in Warsaw, Poland, thanks to my company N47.

My motivation to choose this conference was the fact that I will gain new knowledge, and exchange practical ways of using frontend frameworks. Despite this, given the fact that there were great speakers from the IT world, I had no doubt choosing this tech event. Duration of the event was three days, one workshop day and two speaking conference days.

WHICH WORKSHOP DID I ATTEND TO?

As I was experienced with Vue.js, I wanted to upgrade the knowledge with Nuxt as their workshop description was “It may take it to the next level, thanks to its convention over configuration approach.” I got a certificate of attendance and completion of “My first Nuxt.js application” by the Vue.js Core Team member Darek ‘Gusto’ Wędrychowski. Coding under the eye of ‘Gusto’ and having a wonderful panorama view of Warsaw in my horizon, was definitely a day well spent.

WHICH PRESENTATION DID I ATTEND TO?

Rich agenda with scheduled talks, thoughts about which ones to choose, moreover similar questions were going through my mind. I attended the ones that caught my eye and were mostly within my interests.

At the beginning of each day, there was a high valued speaker opening the day with their talks. The first day I had to meet and listen to the very appreciated, Douglas Crockford with his JSON Saga.

The second day, there was Minko Gechev, a Google engineer working on the Angular framework with the talk ‘The Future of Front-End Frameworks’.

Some other topics that I attended to were about the state management in a world of hooks, some optimizations of the modern JavaScript applications and loading them instantly, as well as Angular and Vue.js 3.0 topics.

WHAT CAUGHT MY MIND?

Two of my favourite talks were ‘The JSON Saga’ – Douglas Crockford and ‘Vue 3.0 for Library Authors’ – Damian Dulisz.

The JSON Saga

Douglas was retelling the story about how he discovered JSON (JavaScript Object Notation). He explained how he did not invent, but found it in the early 2000s, named it and described its usefulness. JSON is a format for storing data and establishing communication between the servers. He explained how some companies complained and did not want to accept JSON because they were used to XML, and could not consider anything else, at that moment. He mentioned that some of the people denied its usage because of it not being a standard. So, what he did next was buying JSON.org, a website which after a few years spread among the users. After a while, JSON got the support of all languages. He announced that there will be no more changes to JSON because for him there is no feature more important than the stability of JSON.

Vue 3.0 for Library Authors

Getting more in details about this topic and Vue 3.0-alpha version will be covered in my next blog.

THE CULTURE AND ENVIRONMENT IN THE CONFERENCE

Frontend Connect was happening in the theatre of the Palace of Culture and Science in Warsaw, Poland where the history and modern world meet at the same time. It is one of the symbolic icons of Warsaw and the place of the city`s rebirth. There were people from all over the world, and the atmosphere was really friendly. Everybody was discussing the topics and shared their work ethics.

CONCLUSION

Visiting conferences is a really good way to meet new friendly people that you have a lot in common with, as well as having an opportunity to reach out to the speaker if you enjoyed the talk, and discuss what you found interesting. We should always strive for more experiences like this and face new challenges within modern technologies. With that being said, we need to nurture our idea to reach our full potential, in order to make a bigger impact in the IT world.