Particle Swarm Optimization (PSO) algorithm is a nature-inspired meta-heuristic that has been utilized as a powerful optimization tool in a wide range of applications since its inception in 1995. Due to the flexibility of its parameters and concepts, PSO has appeared in many variants, probably more than any other meta-heuristic algorithm. This paper introduces the Generalized Particle Swarm Optimization (GEPSO) algorithm as a new version of the PSO algorithm for continuous space optimization, which enriches the original PSO by incorporating two new terms into the velocity updating equation. These terms aim to deepen the interrelations of particles and their knowledge sharing, increase variety in the swarm, and provide a better search in unexplored areas of the search space. Moreover, a novel procedure is utilized for dynamic updating of the particles’ inertia weights, which controls the convergence of the swarm towards a solution. Also, since parameters of heuristic and meta-heuristic algorithms have a significant influence on their performance, a comprehensive guideline for parameter tuning of the GEPSO is developed. The computational results of solving numerous well-known benchmark functions by the GEPSO, original PSO, Repulsive PSO (REPSO), PSO with Passive Congregation (PSOPC), Negative PSO (NPSO), Deterministic PSO (DPSO), and Line Search-Based Derivative-Free PSO (LS-DF-PSO) approaches showed that the GEPSO outperformed the compared methods in terms of mean and standard deviation of fitness function values and runtimes.