This report summarizes the development of a standardized scorecard for evaluating the openness of academic publishers. The assessment was completed in January 2018 as part of the Open Science and Research Initiative of the Finnish Ministry of Education and Culture. The project mapped and evaluated the openness of selected major academic publishers: Association for Computing Machinery (ACM), American Chemical Society (ACS), Elsevier, Institute of Electrical and Electronic Engineering (IEEE), Lippincott, Williams & Wilkins (LWW), Sage, Springer Nature, Taylor & Francis, and Wiley-Blackwell. The dimensions of publisher openness were summarized in a scorecard of seven key factors, providing a new tool for systematic and standardized evaluation. We used data from the publisher websites to compare the key factors of openness, and the publishers were given a chance to provide comments on the collected information. As complementary sources, we utilized data from commonly acknowledged, open databases: Directory of OA Journals (DOAJ), Gold OA Journals 2011-2016 (GOAJ2), Scopus (title list + Scimago), and Sherpa / Romeo. The main results include the scorecard and the evaluation of openness of the selected major academic publishers. These are based on seven key factors: (i) Fraction of open access (OA) journals and their articles of the total publication output, (ii) costs of OA publishing (article processing charges, APC), (iii) use of Creative Commons (CC) licensing, (iv) self-archiving policies, (v) access to text and data mining (TDM), (vi) openness of citation data, and (vii) accessibility of information relating to OA practices. To take a look beyond the publisher level into journal level practices we also sampled individual journals. We use the samples to discuss the distribution of journals according to APCs, their licensing and three impact metrics (CiteScore 2016, Scimago Journal & Country Ranks (SJR) 2016, and Source Normalized Impact per Paper (SNIP) 2016).