The advancement technology of artificial intelligence and e-commerce has increased and this has called for new ways to improve last-mile transportation, which is regarded as an essential part of the logistics value chain, especially in smart logistics. This paper addresses the problem of developing effective routes for autonomous drones in last-mile logistics using deep Q-learning. This paper aims to improve the process of delivery by utilizing the flexibility and intelligence of self-driven autonomous drones in smart logistics transportation. The key challenge for the effective provision of last-mile delivery services remains the decision on the routing of many aerial drones in an indoor urban environment, concerning the restrictions of a time window for delivery, energy consumption and traffic. This paper implements a deep Q-learning paradigm that allows drones to relearn their flight paths and delivery strategy during the lifecycle, thereby reducing the cost in the long run while using the costing strategies as part of the re-engineering process. The approach has been validated through extensive experimentation and simulations. Results obtained indicate that the delivery drones modified for the study attained the designed requirements of deep Q-learning, including optimal navigation and performance that attained 12.8% shorter delivery time, an increase in energy efficiency by 8.4%, and a route quality improvement of 20.1%. Furthermore, highlights the performance of the system in various situations where deep Q-learning and standard routing approaches are compared. This paper not only aids in the minimization of the last-mile delivery constraint by the use of shipping drones but also emphasizes the capacities of reinforcement learning strategies such as deep Q-learning in tackling the routing problems in smart logistics systems. At last, it advocates carrying on deeper into the application of reinforcement learning in the solving of complex optimization problems in various other fields.