short answer: yes
long answer: probably
Let me explain a little bit further.

What is the Angular CLI?

The Angular CLI is a set of of tools provided by the Angular team to make it easier to use Angular (we're specifically talking about versions 2+ here).
The CLI starts by scaffolding out your project and components, then runs your application with a live refresh loop to make development faster, and finally it will help you build and deploy your application.
You can do all of these things yourself, but there's actually quite a bit of setup and configuration involved with building any modern web application. You'd have to touch:
  • TypeScript's .tsconfig, tslint.json
  • your IDE's: .editorconfig
  • Git's .gitignore and .gitkeep
  • Protractor's protractor.conf.js
  • Karma's karma.conf.js
  • Node's package.json
  • Webpack / Bundler configuration

AND you'd need to start building your own
  • App Component, css, html, test
  • App Module
  • Environment Files
  • Index.html
  • main.js
  • Polyfills

But because you typed ng new, the CLI gave you some pretty good defaults out of the box for all of these things.

So it's faster to get started, where's the catch?

The Angular CLI is a bit of a black box. The CLI doesn't give you control of your webpack configuration, it doesn't give you control of what templates / blueprints to use when scaffolding out new components.
That's basically it.
You are sacrificing control for ease-of use.
This should actually be the biggest determining factor for whether or not you use the CLI. Does your project need custom webpack configurations, will you need to pull in additional bundler plugins, or even switch bundlers? Will you need some of the more advanced optimizations of SystemJS or Rollup or Closure Compiler?
If you think you are going to need any of those things, relying on the CLI's build system won't help you, although you could still use its scaffolding capabilities.

How will choosing the CLI affect my project's future?

Today the CLI comes in several parts in an effort to increase the re-usability of the CLI.
A great example of this is the @ngtools/webpack plugin. This is a WebPack plugin that the CLI uses under the hood. The nice thing about this modular approach is that if you can't use the CLI, you can still leverage all of the work they are doing with regard to code splitting, lazy loading, AOT, and more, as long as you are using webpack. At some point in the future there will likely be more independent packages and tools for developers who want to avoid the CLI.
This means that even if you wanted to switch away from the CLI at some point in the future, you should still be able to rely on some of these tools. The CLI team is trying to make self-contained tools that make working with Angular easier for all developers.