The phenomenon of enhancement of projects’ requirement specifications has four main characteristics, each of which increases SE essence difficulties. Newer software products are characterized by:

• Increased algorithm complexity, a greater number of parameters and variables, more complex relationships between variables, and more complicated calculations.
• Growing interface requirements – namely, software-software, software-firmware (software embedded in electronic equipment) and firmware-firmware. We refer here to standard requirements from new software packages to provide a wide range of interface capabilities according to international standards, required interface with leading software packages, and required interface with firmware embedded in the equipment by principal manufacturers.
• More requirements for intra-organizational integration of individual systems into one integrated system. Representative examples of such integrations are the ERP (Enterprise Resources Planning) and CRM (Customer Relationship Management) software systems, which combine the functions of several intra-organizational software systems.
• Increased requirements for inter-organizational integration abilities of software systems. Typical higher SE essence difficulties of this type evolve in the development of software systems for supply chain management (SCM) services. Another area of inter-organizational integration is initiated by governmental agencies for the operation of systems in the area of taxation, which serve both the public and government agencies.

Linda Northrop [2] describes the enhancement of requirements: “current trends are leading us to systems of unthinkable scale in not only line of code, but in the amount of data stored, accessed, manipulated and refined, the number of connections and interdependencies, the number of hardware and computational elements, the number of system purposes and user perception of these purposes, the number of routine processes, the number of interactions and emergent behaviors, and the sheer number of people involved in some way”. She explains the increase in projects’ requirements as follows: “our global appetite for complex software and software-intensive systems continues to increase at a rate comparable to the increases in the computational capacity of hardware”.

The evolution of software systems, in both size and complexity, can be seen by analyzing the growth of the number of software lines of code from a software system’s release to its subsequent release. Sue and Neamtiu [17] examine the size growth of a sample of 7 software applications over several years. The results are shown in Table 1.

Table 1

Application size evolution.

Similar findings were reported by Robles et al. [18]. They analyzed reasonably large and representative samples of stable software systems, large in the number of lines of code, with an active community and user base, and also studied the software systems’ evolution. Most of these software systems show a clear linear growth pattern.

An alternative way to study the evolution of software size and complexity is by examining the number of modules in subsequent software system releases. Turski [19, 20] discusses Lehman’s laws of software evolution. In [19], Turski presents a typical example of software system growth over 21 releases.

In the first release, there were 977 modules, while in the 21st release there were 2,315 modules; this means an average growth of 70 modules in the number of software modules, compared to its former release, and about a 137% growth rate over 20 releases. Similar results of steady growth were found for software systems based on open source software, specifically related to the Linux kernel and its many releases over 14 years ([21, 22]). The evolution of software complexity is also discussed by Arthur [23]. He proposes three causes for software systems’ growing complexity: an increase in the diversity of the “species” served by the system, an increase of structural sophistication, and additional functionality to overcome performance limitations.

The continuous trend of decrease of project time schedules has been typical of all types of software projects over recent decades. It is the general belief that project schedules of the same magnitude are cut by 50% every 2-4 years. The implications of this trend on SE capabilities to handle SE essence issues are severe:

• In general, shorter project time schedules means less time available for reviewing and testing. Thus, the ability to fully solve SE essence difficulties is reduced.
• The requirement to complete projects within ever shorter time constraints leads to a need to employ a larger development team. Larger development teams, in turn, eventually causes a need for more cooperation and coordination efforts, which is harder to achieve.
• Furthermore, shorter time schedules, in many cases, forces the developer to use partners or subcontractors or outsourcing services for parts of the project. These types of project organization require higher control efforts and cause coordination difficulties. Thus, this situation is involved with higher risks for software essence errors.

The increased frequency of customer demands for changes of requirements specification during the development process and after the project has been completed, typical to the last decade, yields lesser ability to cope with SE essence difficulties:

• The handling of more changes of requirements specifications during software development, adds to the project’s SE essence difficulties. It increases the work burden and time pressures on the project teams, leading to inability to perform all the required reviewing and testing of the software system. As a result, a project that was involved with many changes is expected to be characterized by high rate of unsolved SE essence difficulties.
• Demands for changes after the project was completed usually create SE essence difficulties that are harder to solve, as they are usually performed by less experienced team.

The severe negative effect of a higher frequency of changes of the requirementד is discussed by Berry [3], Williams et al. [24] and Rahman et al. [25]. Berry argues that “a typical software development method is effective in its first application development problem. However, once developers have built and developed a version with its method, the requirements begin to change, whether from E-type system pressures or client and user demand. When an inevitable change comes along, modifying the method, documenting artifacts is so painful that developers avoid doing it the right way, by carefully tracking the change’s effects. Instead, they create a quick patch that increases the system’s brittleness”. Williams et al. [24] and Rahman et al. [25] refer to the risks involved in late changes of requirements of schedule delay and additional costs, which are harder to maintain in the future. On top of these impacts, a severe risk discussed in these studies is that of the “snowball effect”, which relates to additional changes in other parts of the project needed on top of the original initiated change. The original change task is completed only once the entire series of additional changes has been satisfactorily completed. They suggest a risk assessment study prior to a decision to perform a requested change of requirements.

The IT advancements, especially the introduction of integrated software systems, inevitably create growing reliance and dependence of users on these systems, for the operation, control and management of organizations. Thus, organizations became less and less tolerant to SE essence errors, even to minor ones. Organizations that once could tolerate an error that paralyzed the operation of a minor application for a short time, are no more able to tolerate it. Due to systems’ integration, any error affects now the whole integrated software system. This trend raises the need for higher perfection of the software development process, which naturally adds to the difficulty of solving the SE essence difficulties.

Mens et al. [26] describes the dependence on software as general: this dependence “takes place in all sectors of society, including government, industry, transportation, commerce, manufacturing and the private sector”. They claim that the low quality of software and specifically software maintenance are the challenges of SE, and list relevant areas of research and development for SE. supported by other researchers’ findings, Duggan [27] discusses the increased reliance of business on IT and lists several reasons: (a) “To support missions and priorities, either for strategic enhancement or competitive necessity. This accentuates the challenges that face IS developers”; (b) “Organizational drives to apply more sophisticated technologies and establish flexible communication networks to accommodate a variety of data and processing distribution strategies, multimedia operations and integrated systems”; and (c) “Increased security concern that now attends the greater movement of data”.

The use of purchased or reused software, in its variety of forms, is probably a very effective, if not the most effective, response to growing SE essence difficulties. These SE practices include purchase of software packages (COTS software), purchase of software components for their integration in required applications, as well as software reuse. Common to all these courses of action is their reliance on software that has already been tested and corrected, according to defects identified in previous tests and by earlier users. It should be noted that applying COTS software and software reuse do not completely eliminate SE essence difficulties for the development teams. These teams still have to cope with SE essence difficulties related to the analysis of the requirement specifications and their fit with regard to the proposed software package. Still, the major part of the analysis, design and testing of SE essence specific difficulties is saved. Applications of these practices can cause: (a) substantial reductions of the required development resources; (b) A lower rate of SE essence errors; (c) A shorter project schedules and better schedule keeping; and (d) A smoother development process, due to increased standardization, resulting from repeated use of the same software package components.

The application rate of COTS software, purchased software components, and software reuse is still relatively low compared to their evaluated potential. The level of software reuse has been continuously growing over the last decades, as presented by Jones [30] for the period 1955-2005 and as predicted for 2015. See Table 3 for reuse percentages.

Table 3

Software Reuse - 1955-2015.

In their 1995 paper, Garlen et al. [31], while believing that the future of software development productivity depends on software reuse, discuss the difficulties in realizing the reuse potential. They claim that the main reason for difficulties in matching reused software in a new software project is the incompatibility in programming language, operating platform and database scheme. More than ten years later, in 2009, Garlen et al. [32] and Andersen [33], facing the still low implementation of software reuse, once again discuss the mismatch issues of software reuse as the reason why the potential of software reuse is still far from being realized. A variety of methodologies to extend software reuse, including the building of software reuse infrastructure, prioritizing of candidate reuse projects, and establishing software reuse libraries, are offered by Sutcliffe et al. [34] and Sherif et al. [35]. Another way to promote reusable software discussed by Augustwamy and Frakes [36] is through the designing and building of reusable components that will be easier to apply. In an empirical study, he examines design principles that could guide the developers of reusable software components.

CBSD (COTS Based Software Development) is considered to be the next revolution in software development, with the advantages of lower costs, shorter timetables and higher quality. It is also expected to contribute to easier maintenance and be based on replacement with new enhanced COTS components [37]. Much research has been dedicated to analyzing the current challenges of CBSD in the finding and selection of appropriate COTS product implementation. The difficulties of selection and systematic identification of the needed COTS products are considered the main barrier to wider implementation of CBSD. Methods for improving the selection process are discussed by many authors, to mention just few [37-43]. The documentation quality of COTS components, namely incomplete and inaccurate documentation, as well as communication with vendors, are mentioned as difficulties in the integration and testing stages of implementing CBSD projects [38], and [44]. Another reason preventing wider implementation is the expected high efforts needed to select and identify the appropriate COTS components that cause developers to prefer self-development. The application of COTS implementation efforts’ estimation method can promote COTS use [45].

It is expected that over the next few years, applications of new or improved computer-aided SE (CASE) tools will improve general software development performance. Today’s CASE tools enable teams to automate segments of the software development process with no defects; specifically, to automate the design generation out of requirement specifications, and the automated code generation out of design. Other CASE tools provide updated and accurate documentation and support coordination among members of large development teams. Thus, while the current CASE tools mainly contribute to solving accidental SE difficulties, they also contribute, to some extent, to reducing SE essence difficulties by reducing rates of design-generated errors, improved documentation and coordination. The success rates of CASE applications in the nineties were disappointing. Research results identified the main causes for low implementation as follows: they were not user-friendly, offered low management support, and a lack of voluntariness [46]. This low implementation rate is still discussed a decade later [47]. Much research effort has been dedicated to the development of methods to overcome the difficulties of wider implementation of CASE tools (some typical examples, [48-51].

The improvement in testing techniques, especially by automated testing, as well as improved development tools and design reviews, have improved SE capabilities by reducing the defect rates and saving correction efforts. Based on Jones’s [30] results, Table 4 presents the defect rates in the US for the years 1955-2005 and a prediction for 2015.

Table 4

Average defect rates in US software projects, 1955-2015.

The higher percentages of defects removed and the lower numbers of defects delivered helped SE cope with users’ growing dependence on the regular operation of software systems and their lower tolerance to SE errors.

Automated testing provides for a better error identification rate compared with manual testing, as related to SE accidental errors as well as SE essence errors. Currently, automated testing tools do not serve all types of software. Moreover, the planning and defining of the automated testing plan (the scenarios) still requires numerous manual resources. Thus, surveys of testing methods application find that, although the contribution of automated testing is already substantial, it’s still far below its full potential [52-54]. Research efforts into the development of tools for automating software testing include the development of tools for specific programming languages, the analysis of organizational aspects, as well as economical aspects [55-59].

Open source software (OSS) provides software developers with free access to a great variety of ready-to-use software program codes, while allowing one to make changes and adaptations to suit individual needs. Similarly to the application of COTS software, the application of OSS - which others have already developed, tested and corrected - presents a sizeable potential for reducing the required development resources. It should be noted that applying OSS still requires the developers to handle SE essence difficulties involved in analyzing the fit of the proposed OSS package to the customer’s requirement specifications and in performing the necessary adaptations, Still, the major part of the analysis, design and testing of SE essence difficulties are saved. It is expected that software development, based on OSS, will become one of the major answers to the challenges of growing SE essence difficulties. A technical and economical evaluation should lead to the adoption of OSS [60].

Growing application rates of open source software OSS, due to the development of culture and institutions dedicated to OSS implementation [61-63] have been noted in the last few years. Case studies of OSS development applying open source tools are encouraging, despite the identified interoperability and adaptability problems and other limitations of these tools [64]. The accumulation of open source components leads to the growing difficulty of maintaining open source libraries and repositories, especially the categorization of components. The development of tools for maintaining these libraries, including the automatic categorization of components, are the focus of current research projects [65].