A new automated smart cart system for modern shopping centres

Received Dec 31, 2020 Revised Mar 25, 2021 Accepted Jun 14, 2021 Modern shopping centres are undoubtedly a beehive of intense shopping activities. However, customers are often plagued by salient challenges, which may include fatigue derived from pushing trolleys around the mall and prolonged sorting of bills by the cashier. These shopping challenges could be daunting for the elderly, disabled, pregnant and nursing mothers. In this paper, we addressed these shopping challenges by developing an autonomous shopping cart with the following characteristics; (1) it follows the customer’s movement relieving the need to push a cart, (2) it bills automatically all stock placed in the cart, (3) it prompts the customer to make payment and updates each stock via a local database. Our design adopts a Raspberry Pi, a camera and a few direct current motors programmed to achieve autonomy. We used an open-source cross platform software called XAMPP to create the database and used RFID tags to bill the items placed in the cart automatically. The system updates payments and communicates these transactions to a local database via nRF24 wireless transceivers. The experimental tests conducted demonstrate that our system successfully followed customers accurately within the mall. We consider our design a major contribution to the vision of automated shopping systems for the near future.


INTRODUCTION
In modern shopping centres (malls and super markets), shopping carts are often seen as plastic or metal basket-like frames on wheels pushed by customers to carry items or products around the centre [1], [2]. However, shopping via traditional shopping carts in modern centres may pose several challenges to customers including the burden of pushing heavy carts by the elderly, physically challenged, pregnant and nursing mothers. Other salient challenges may include delay at long queues, prolonged sorting and billing details at the till. Furthermore, mall managers often need to track manually each stock available and sold in the large centre, which can be burdensome. These challenges require innovative solutions, which served to motivate the ideas demonstrated in this paper.
Several authors have proposed different approaches to address the above challenges in the literature. However, these efforts focused more on autonomous mobility of the shopping cart based on the line-following approach [3]- [7]. Image processing has found application in many fields among which include [8], [9]. Its used for autonomous mobility of shopping cart based on some image colours for tracking object as well as some android application-controlled approaches were presented in [10]- [14]. While, other efforts addressed automated billing system issues and developed predictive algorithms to suggests items to customers [15]- [17]. A multifunctional automated trolley system based on an android application controlled via a smart phone fixed to a trolley was presented in [2]. Their system achieved good performance. However, it required a predefined black marked line in order to guide the cart's movement. These predefined markers typically constrain the customer's movement thus limiting the flexibility of their design. Similarly, authors in [3] used a line-following robot to drive a shopping cart. They deployed an ultrasonic and line sensor to track the customer's movement and an android application to guide the customer to where goods were located in the mall. They used also an RFID localization scheme and an android application to track the trolley's location. Nevertheless, the line following approach in [3] limits the flexibility of their design similar to [2]. Authors in [4]- [6] used ultrasonic sensors to detect obstacles and to track customers. They used Arduino integrated development environment (IDE) to drive a system of motors to provide autonomous mobility to the cart. Their use of ultrasonic sensors may be similar to other use-cases found in [18], [19]. However, their designs in [4]- [6] are typically unable to differentiate actual users from other stray carts that infiltrate the range of the ultrasonic sensor. A mobile following robot capable of avoiding obstacles, while tracking user was developed in [7]. By using preprogramed images of different objects, authors demonstrated that their system could successfully track customers. A similar concept was presented in [20] that could navigate and avoid obstacles. Authors used this concept to develop autonomous shopping carts. Authors in [21] developed an algorithm to search and map the position of a cart user. They used laser range sensors to estimate a user's position and distance, a gyro-sensor and a rotary encoder. However, their system's accuracy depends strongly on the mapping algorithm, which can be often slow due to complex computations.
Kim et al. [22] investigated different human perspectives toward the design of a convenient and efficient smart cart system. They used a questionnaire and survey method to investigate the best position of an RFID tag and reader, touch screen, a barcode system and a cart module in the smart cart system. The results obtained showed that most customers prefered the touchscreen to be located at about 250 mm from the handle of the cart, while the cart handle should be tilted at 45 degrees. A similar feasibility study was conducted in [23] to develop an intelligent cart system for use in airports. Authors used the ant colony clustering (ACC) algorithm to find quasi-optimal positions to assemble the cart. The waiting time and precise route of the carts were determined via simulation and detailed procedures were used to compute a sequence of control commands to move the cart. Authors provided results to show that their approach aligned and arranged scattered carts automatically. Marchetti et al. [15], a data fusion algorithm based on particle filter was developed to combine information from an RGB-D camera and a planer laser range finder to estimate and detect the position of a user. These were incorporated into a proposed autonomous shopping cart to track users. To estimate the user's position, authors in [15] adopted a k-clustering technique. They demonstrated that their system consistently tracked the users. Prerana et al. [16] developed an approach that used an RFID reader, which automatically scanned and billed products dropped in the cart. The system added the cost of an item automatically to a list of billed items once the RFID reader scanned it. The customer would activate a switch button to detect the product code in order to remove items no longer required. The system transfers the final product bill via zigbee to the counter where payment can be made. Despite demonstrating the capability of autonomous billing, the system in [16] fails to incorporate autonomous mobility and automatic payment systems in their model. The approach in [17] used machine learning technique to enable a robot to learn and modify its knowledge about different customer shopping preferences. The system logged a repository of digital data about the user's cultural preferences into a database. It used this information to profile each user and to predict each user's preference. In their design, an inbuilt camera captures images used to train the system to identify the user's emotional status. The results obtained demonstrated the feasibility of their approach. However, their approach is relatively complex because it requires large processing capabilities, which makes it costly for real time realization and implementation.
Inspired by past efforts, we provide in this paper an encompassing design of an automated shopping cart system that improves flexibility, provides instantaneous billing and automated inventory tracking via a local database. The proposed new automated smart cart system used a simple combination of an ultrasonic sensor, Raspberry Pi, some direct current (dc) motors, a video camera, RFID tags and readers, an nRF24 wireless transceiver, an LCD display, a GSM module and a database. Our design provides a low cost innovative smart shopping cart that tracks accurately the customer, provides automatic billing, and displays each unit cost and the total cost of each item on an LCD screen. Furthermore, our system keeps track of all inventories for shop managers/owners. We improved on the approach proposed in [5], [6] by using ultrasonic sensors instead of the line-following technique. We improved also the billing system over the approach in [16] by providing a notification system along with using a local database to keep track and update of stocks in the mall for the shop owner. In addition, we conducted a detailed analysis to examine the performance of our system and its suitability for commercial production. The remainder of the paper is structured as shown in: section 2 presents the research method of the proposed smart innovative shopping cart system. Section 3 presents the experimental results and discussion while section 4 concludes the paper.

RESEARCH METHOD
We adopted a three-stage methodology in our design namely, (1) to develop the autonomous smart cart system (2) to build the billing system and (3) to design the database modules. A schematic of our design is shown in Figure 1 and we explain each stage as:

Autonomous smart cart development
The autonomous smart cart consists of a reinforced aluminium sheet frame supported on a two wheel and a rear-supporting wheel. It contains a basket-like opening, where customers could drop items while shopping. A central processing unit (Raspberry Pi) coordinates and controls the movement of the cart. To provide autonomous mobility to the cart, we adopted an image-processing algorithm based on the RGB colour-processing model [24]. The algorithm acquires an object frame via a camera installed in the top front of the smart cart system, which makes up the video acquisition unit as shown in Figure 1. Similar to the threshold computation approach presented in [25], the algorithm compares each newly acquired object frame with pre-saved object frames in the system and then computes a threshold pixel level using (1).
where, 2 2 2 , G & RB are the current object image pixel acquired, while 1 1 1 , G & RB are the pre-saved background object image pixel. If the computed threshold level is less than 500pixel value, the system assumes that the user is found. Else, if the computed pixel value is higher, then the user is assumed not to be within the range of detection by the smart cart. The system adjusts always the location of the pixels computed after detection to the centre of the camera. It uses this adjustment to direct the cart to go either left or right for the horizontal axis and to go either forward or standby for the vertical axis. This guarantees that the user is continuously tracked.
The system acquires two-dimensional object images (x, y) through the camera to determine continuously the position of the user. We used an object-image window size of 320 by 240 pixels to determine the user's position. Usually, if the computed horizontal axis (x-coordinate) of the image is less than 140 pixels, the cart detects the user to be at the right, but if it is greater than 200 pixels, then the cart detects the user to be at the left side due to a 180-degree image inversion of the camera. Similarly, if the computed vertical axis (y-coordinate) of the object image is greater than 170 pixels, the cart assumes that the user is a little farther away and triggers the motor incorporated in the driver unit to move the cart. Else, if otherwise, the cart remains in the standby mode. We provide a summary of this image-processing algorithm in Algorithm 1. The driver unit comprises of a motor driver integrated circuit (L298N), two 12 volts dc motors and a battery. The output of the video acquisition and processing unit trigger the motor driver via the Raspberry Pi processor. We labelled the two dc motors as motor 1 and 2, respectively. Each motor had two terminals labelled as A and B and controlled by two logic states 1 and 0. The logic 1 signifies a high (on) state, while logic 0 signifies a low (off) state. When motor 1 and 2 received a logic combination of 1 and 0 at terminal A and B respectively, it implies that the cart will move in the forward direction. If any of the terminals is reversed for each motor, then the cart will move either left or right direction. However, if all the terminals are at logic 1 or 0 for both motors, it implies that the cart will be on standby mode, i.e. in a stationary position. We summarize in Table 1 the movement pattern of the proposed cart based on the logic received from the Raspberry Pi. Note in Table 1, that N/A denotes that the coordinate values are not applicable to the movement pattern of the cart.
In the obstacle detection unit, an ultrasonic sensor, which is a low-cost transceiver, was utilized. The ultrasonic sensor has a separation distance of 2 cm between the transmitter and receiver, which was used to approximate the distance between the transmitted ultrasound signals at 40 Hz from the cart. This determined the time taken to receive the reflected echo pulse from the obstacle along the path as shown in Figure 2. We used an HCSR04 ultrasonic sensor to measure the distance between ranges of 2-400 cm. If the measured distance by the ultrasonic sensor using (2) is less than the set threshold value (20 cm), the Raspberry Pi triggers the cart to remain in a standby mode since an obstacle apparently exists in its path. Other details and configuration of the principle of operation of the ultrasonic sensor used for the obstacle detection can be seen in [18]. We used an open source processing software environment to program the Raspberry Pi.  where is the distance computed in (cm), ℎ is the echo pulse width high time, and is Sound velocity (m/s). The sound velocity is 340m/s or 29microseconds per cm.

The billing system
This module comprises of an RFID reader operating at a frequency of 125 kHz, incorporated inside the basket-like opening in the smart cart. We tagged each item using a unique ID in all our experiments. This enabled the system to scan automatically each item placed in the cart. Usually, the RFID reader would read these IDs via the serial pin of the Arduino Uno board. The Arduino board controls the billing module, as information received from the transmitter (RFID reader) is sent to the developed database as shown in 2.3. It achieves this via the nRF24 wireless transceiver, interfaced to the Arduino using a serial peripheral interface  (SPI). We covered the outer part of the smart cart with a radio frequency (RF) shield to prevent the RFID reader from scanning other products outside the cart. The system extracts from a database the name and cost of each item placed in the cart and displays these on an LCD. As the items placed in the cart increases, the cost of each item is added to the cost previously extracted from the database and this is displayed on the LCD. When a user changes his/her mind about an item and removes it from the cart, the RFID scans automatically the item and communicates its ID to the database, which updates automatically the list of intended purchased items and their corresponding cost. We summarize this entire process in Algorithm 2.

Database module
This section describes the local database that contains information about each item stocked in the mall. We created the database using the MySQL and the XAMPP control panel, which is an open source cross platform. We used the MySQL and Apache from the XAMPP control panel in our design. The domain "localhost/phpmyadmin/" was launched via the address bar of a Google Chrome browser. We named the database as "shopping cart and table" and displayed this on the phpmyadmin interface, which indicates that the database was created successfully. The phpmyadmin interface launches the created "shopping cart and table" database. The "Database Shopping Cart" was assigned. Five different fields namely; (1) serial number (s/n), (2) tag ID, (3) item name, (4) item price, and (5) amount in stock. The data type for each field was assigned and the data format type called "integer" was assigned to S/N and to amount in stock, while the variable character is assigned to all other fields. Further details concerning the creation of the database with other necessary fields using the XAMPP software can be found in [26]. The database was interfaced with an ethernet shield and an Arduino Uno board, which retrieves from the billing module any necessary information requested by the item ID as shown in section 2.2. The retrieved information from the local database is sent to the billing module via the ethernet shield and the nRF transceiver. The overall system architecture comprising of all the three modules is summarised in Figure 3.

RESULTS AND DISCUSSION
We tested the new smart cart system in two stages. First, we tested separately the individual modules of the proposed system viz the autonomous module, the billing and database module. Secondly, we tested the communication system between all the three modules as well as the entire system. In testing the autonomous module of the proposed system, the camera feed rate was set to 30 frames per seconds, while the best orientation of the camera that gives better result was investigated as summarized in Table 2. Observe that an average response time of 33ms was required to trigger the autonomous tracking of the user. We designed the billing system to scan an item at a time, particularly as each item was dropped in the cart. The system had a delay of 500 ms after scanning automatically the item placed inside the cart. This delay was necessary in our design to enable the system to communicate with the local database in order to fetch the cost of each item, compute and sum the total bill of items in the cart. This was initialized by the pre-saved colour of an object that will usually be attached to the user's body. We measured this response time only when the user is within the range of operation of the proposed system, which is in the range of 0 to 475 cm. The system becomes non-responsive at ranges greater than 475 cm as shown in Table 2. The system remained in a standby mode when the user was positioned within a range of 0-50 cm. In this case, it did not matter whether the user was positioned directly in front of the cart or on its sides. In addition, the system performed well in terms of successfully tracking the user, especially when the user was positioned directly in front of the cart. However, it performed poorly when the user was positioned at the bottom plane as shown in Table 2. Generally, an average-response scan time of about 300 ms after 60 trials was recorded for each item as shown in Table 3), with a corresponding bill computation time of less than 850 ms at varying distances between 1 to 5 cm. Nevertheless, the system may remain non-responsive and thus continue to scan indefinitely when the scanning distance is greater than 5cm as shown in Table 3. The screen shot of the created database structure and its fields are shown in Figure 4. We provide also in Figure 5 an image of the physical prototype of the cart system.

CONCLUSION
This paper has presented the design and prototyping of an innovative smart shopping cart system. The autonomous capability of the system was achieved using pre-saved object images that were processed and used to trigger the mobility of the cart. A Raspberry Pi processor was used to coordinate the cart's movement. In addition, a local database was created using XAMPP software, which contained the stocked items and their corresponding ID, price and the number of stocks. The database keeps track of available stocks in the mall. Furthermore, the billing module of our system communicates via the nRF24 wireless transceiver to fetch information required from the database. This was achieved by scanning automatically the item's ID as they were dropped in the cart. Our test results showed that the cart system tracks and follows the user successfully and automatically particularly when it was triggered by the pre-saved colour image of an object placed on the user's body. Our system also billed items automatically that were dropped in the cart. It similarly updates successfully the stock on the database. Future works will consider how to integrate in our design an online payment unit in addition to providing automatic battery charging and cart parking capability, this will further improve the efficacy and performance of the system.